Changeset 0.36.0 (#212)

This commit is contained in:
2026-03-20 16:08:12 +00:00
parent 668c608b4f
commit 0ab2800c7e
13 changed files with 11598 additions and 1442 deletions

View File

@@ -0,0 +1,810 @@
# DESIGN-0.32.0 — Multi-Replica HA
Run 23 backend replicas across nodes for node-level availability.
Five changesets. No sub-versions — the work is tightly coupled and
each CS builds on the previous.
Depends on: v0.31.2 (current HEAD).
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
leader election for the task scheduler. Every replica polls; PG
serializes the claims. Extends to all shared mutable state — tickets,
rate counters — keeping PG as the sole coordination layer. No Redis,
no K8s API dependency, no new infrastructure.
---
## What Already Works Multi-Replica
These require zero changes:
- **REST API** — stateless, JWT auth, any replica serves any request.
- **PG + S3 + CephFS** — shared storage infrastructure.
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
`broadcastHook``pg_notify` → remote pod `publishLocal()`
local WS subscribers. Fully wired, just never tested at N>1.
---
## What Needs Work
Five areas of in-process mutable state that break at replica count > 1:
| State | Current | Problem at N>1 | Fix |
|-------|---------|----------------|-----|
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |
---
## CS0 — Schema: `020_ha.sql`
New migration for both PG and SQLite. No changes to existing 019
task schema — `next_run_at` nullable already supports the claim
mechanism.
### PG: `server/database/migrations/020_ha.sql`
```sql
-- ==========================================
-- Chat Switchboard — 020 Multi-Replica HA
-- ==========================================
-- Shared state tables for multi-replica operation.
-- v0.32.0: ws_tickets, rate_limit_counters.
-- ==========================================
-- =========================================
-- WEBSOCKET TICKETS (replaces sync.Map)
-- =========================================
CREATE TABLE IF NOT EXISTS ws_tickets (
id TEXT PRIMARY KEY, -- 128-bit hex token
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
ON ws_tickets (expires_at);
-- =========================================
-- RATE LIMIT COUNTERS (replaces in-memory)
-- =========================================
-- Token bucket approximation using time-windowed counters.
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
-- Window is truncated to the second for the configured rate.
CREATE TABLE IF NOT EXISTS rate_limit_counters (
key TEXT NOT NULL,
window TIMESTAMPTZ NOT NULL, -- truncated timestamp (window start)
tokens REAL NOT NULL DEFAULT 0, -- tokens consumed in this window
PRIMARY KEY (key, window)
);
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
ON rate_limit_counters (window);
```
### SQLite: `server/database/migrations/sqlite/020_ha.sql`
SQLite parity — structurally identical but with dialect adjustments.
These tables are functional in SQLite for single-process test parity,
though multi-replica is PG-only in production.
```sql
CREATE TABLE IF NOT EXISTS ws_tickets (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
ON ws_tickets (expires_at);
CREATE TABLE IF NOT EXISTS rate_limit_counters (
key TEXT NOT NULL,
window TEXT NOT NULL,
tokens REAL NOT NULL DEFAULT 0,
PRIMARY KEY (key, window)
);
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
ON rate_limit_counters (window);
```
### Deliverables
- [x] `020_ha.sql` (PG)
- [x] `020_ha.sql` (SQLite)
- [x] CI green on both pipelines (migration runs, tables exist)
---
## CS1 — Task Scheduler: SKIP LOCKED
Replace the read-then-execute scheduler with atomic claim semantics.
Every replica runs the poll loop. PG serializes task handoff.
### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)
New store method. Single atomic statement — SELECT + UPDATE in one
round trip. Returns at most one task per call.
**PG implementation:**
```sql
UPDATE tasks
SET next_run_at = NULL
WHERE id = (
SELECT id FROM tasks
WHERE is_active = true
AND next_run_at <= NOW()
ORDER BY next_run_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING <taskColumns>
```
Mechanics:
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
candidate row, skip it instantly (no wait, no deadlock).
- `SET next_run_at = NULL` — claimed task disappears from future
polls. `advanceNextRun` restores it after execution.
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
- The existing partial index `idx_tasks_next_run` covers the WHERE
clause (`is_active = true AND next_run_at IS NOT NULL` by
implication — NULL rows won't satisfy `<= NOW()`).
**SQLite implementation:**
SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
the current `ListDue` behavior unchanged for SQLite. The store
interface accommodates both:
```go
// TaskStore additions
ClaimDueTask(ctx context.Context) (*models.Task, error)
```
SQLite `ClaimDueTask` just does:
```sql
SELECT <taskColumns> FROM tasks
WHERE is_active = 1 AND next_run_at <= datetime('now')
ORDER BY next_run_at ASC LIMIT 1
```
Then sets `next_run_at = NULL` in a second statement (single writer,
no contention).
### 1.2 — Conditional `CreateRunExclusive`
Belt-and-suspenders: prevent double-execution even if two replicas
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
but defense in depth).
```sql
INSERT INTO task_runs (task_id, status)
SELECT $1, 'running'
WHERE NOT EXISTS (
SELECT 1 FROM task_runs
WHERE task_id = $1 AND status IN ('running', 'queued')
)
RETURNING id, started_at
```
Returns `sql.ErrNoRows` if a run already exists → skip execution.
New store method:
```go
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
```
### 1.3 — Scheduler Loop Rewrite
`scheduler.go` changes:
```go
func (s *Scheduler) poll() {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
if !cfg.Enabled {
return
}
// Claim tasks one at a time until none remain or budget exhausted.
// Each replica claims independently — PG SKIP LOCKED serializes.
claimed := 0
for claimed < cfg.MaxConcurrent {
task, err := s.stores.Tasks.ClaimDueTask(ctx)
if err != nil {
break // sql.ErrNoRows or real error — either way, done
}
claimed++
go s.execute(ctx, task)
}
}
```
`execute()` changes:
- Remove the `GetActiveRun` TOCTOU check — replaced by
`CreateRunExclusive`.
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
But adoption also uses `CreateRunExclusive` semantics (transition
only if still `queued`).
- `advanceNextRun` still computes and sets `next_run_at`, making the
task visible to future polls again.
### 1.4 — Remove `ListDue` from Interface
`ListDue` removed from `TaskStore` interface. Replaced by
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
`CreateRunExclusive` added alongside it.
Updated interface:
```go
type TaskStore interface {
// ... existing CRUD ...
// Scheduler queries
ClaimDueTask(ctx context.Context) (*models.Task, error)
SetNextRun(ctx context.Context, id string, nextRun interface{}) error
SetLastRun(ctx context.Context, id string) error
IncrementRunCount(ctx context.Context, id string) error
// Run history
CreateRun(ctx context.Context, r *models.TaskRun) error
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
UpdateRun(ctx context.Context, id string, status string, ...) error
TransitionRunStatus(ctx context.Context, id string, status string) error
GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
}
```
### Deliverables
- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
- [ ] `CreateRunExclusive` — PG + SQLite
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
- [ ] Remove `ListDue` from interface + both stores
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
disjoint task sets)
- [ ] CI green
---
## CS2 — PG Ticket Store
Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
implementation. The middleware already programs to the `TicketValidator`
interface — the swap is clean.
### 2.1 — Store Interface
New interface in `store/`:
```go
// TicketStore manages short-lived single-use WebSocket auth tickets.
type TicketStore interface {
Issue(ctx context.Context, userID string) (string, error)
Validate(ctx context.Context, ticketID string) (string, bool)
Reap(ctx context.Context) (int, error)
}
```
### 2.2 — PG Implementation
**Issue:**
```sql
INSERT INTO ws_tickets (id, user_id, expires_at)
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
```
Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).
**Validate (atomic consume):**
```sql
DELETE FROM ws_tickets
WHERE id = $1 AND expires_at > NOW()
RETURNING user_id
```
Single statement — delete + return. If expired or already consumed,
zero rows → `("", false)`.
**Reap (TTL cleanup):**
```sql
DELETE FROM ws_tickets WHERE expires_at <= NOW()
```
Called by a system task or inline during poll. No dedicated goroutine
needed — the scheduler's 30s tick can piggyback, or register as a
lightweight system function.
### 2.3 — SQLite Implementation
Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
instead of `NOW() + INTERVAL`).
### 2.4 — Adapter for Middleware Interface
The middleware `TicketValidator` interface is:
```go
type TicketValidator interface {
Validate(ticketID string) (string, bool)
}
```
The store's `Validate` takes a `context.Context`. Thin adapter:
```go
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
type TicketValidatorAdapter struct {
Store store.TicketStore
}
func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return a.Store.Validate(ctx, ticketID)
}
```
### 2.5 — Wiring in `main.go`
Replace:
```go
ticketStore := events.NewTicketStore()
defer ticketStore.Stop()
```
With:
```go
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
```
The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
directly. The WS auth middleware receives `ticketAdapter` (satisfies
`TicketValidator`). No reaper goroutine — use system function or
scheduler piggyback.
### 2.6 — Reap Strategy
Option A: Register `ticket_reap` as a system function
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
infrastructure. Cons: requires task to exist.
Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.
**Recommendation: Option B.** At 50 users, the table will have at most
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
free. Revisit if ticket volume grows.
### 2.7 — Delete `events/tickets.go`
The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
lives in `events/` (or alongside the middleware — whichever avoids
import cycles).
### Deliverables
- [ ] `store.TicketStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
`TicketValidator` interface
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
- [ ] Inline reap in scheduler poll
- [ ] Delete `events/tickets.go`
- [ ] Add `Tickets` field to `store.Stores`
- [ ] CI green
---
## CS3 — PG Rate Limiter
Replace the in-memory token bucket (`middleware.RateLimiter`) with a
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
burst 8), so transaction volume is trivially low.
### 3.1 — Store Interface
```go
// RateLimitStore manages distributed rate limit counters.
type RateLimitStore interface {
// Allow checks if a request is within the rate limit.
// Returns (allowed bool, tokensRemaining float64).
// Atomically increments the counter if allowed.
Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)
// Cleanup removes expired windows.
Cleanup(ctx context.Context, maxAge time.Duration) error
}
```
### 3.2 — PG Implementation: Sliding-Window Token Bucket
The in-memory implementation uses a classic token bucket (refill based
on elapsed time). The PG version approximates this with time-bucketed
counters.
**`Allow` — atomic check-and-increment:**
```sql
-- Upsert the current window's counter and check burst limit.
-- Window = current second (truncated).
INSERT INTO rate_limit_counters (key, window, tokens)
VALUES ($1, date_trunc('second', NOW()), 1)
ON CONFLICT (key, window)
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
RETURNING tokens
```
Go logic after the RETURNING:
1. Query returns `tokens` (count in the current 1-second window).
2. If `tokens > burst` → denied, return `(false, nil)`.
3. Else → allowed.
This is a fixed-window approximation, not a sliding window. For auth
rate limiting at 5 req/s burst 8, the practical difference is
negligible — worst case allows 2× burst at window boundaries, which
is acceptable for this use case. A true sliding window would require
reading adjacent windows and interpolating, adding complexity for
near-zero benefit at this scale.
**Cleanup:**
```sql
DELETE FROM rate_limit_counters
WHERE window < NOW() - $1::interval
```
Called alongside ticket reap in the scheduler poll loop.
### 3.3 — SQLite Implementation
Same logic, `datetime('now')` for window truncation. Single-process,
so the PG atomicity guarantees are naturally satisfied.
### 3.4 — Middleware Swap
Replace `middleware.RateLimiter` struct with one backed by the store:
```go
type RateLimiter struct {
store store.RateLimitStore
rate float64
burst int
}
func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
return &RateLimiter{store: store, rate: rate, burst: burst}
}
func (rl *RateLimiter) Limit() gin.HandlerFunc {
return func(c *gin.Context) {
key := "auth:" + c.ClientIP()
allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
if err != nil {
// DB error — fail open (don't block auth on rate limit DB failure)
c.Next()
return
}
if !allowed {
c.Header("Retry-After", "1")
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
"error": "rate limit exceeded",
})
return
}
c.Next()
}
}
```
**Fail-open policy:** if PG is down, allow the request. The auth
endpoints have their own protections (bcrypt, lockout). Blocking
legitimate logins because the rate limit table is unreachable is worse
than allowing a brief burst.
### 3.5 — Wiring in `main.go`
Replace:
```go
authLimiter := middleware.NewRateLimiter(5, 8)
```
With:
```go
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
```
Delete the in-memory cleanup goroutine (replaced by scheduler reap).
### Deliverables
- [ ] `store.RateLimitStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
- [ ] `main.go` wiring
- [ ] Add `RateLimits` field to `store.Stores`
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
- [ ] Delete in-memory visitor map + cleanup goroutine
- [ ] CI green
---
## CS4 — WebSocket Cross-Pod Delivery
The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
already implemented and handles room-scoped events correctly. The gap:
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
to local connection send channels. Users on other pods are never reached.
### Affected call sites (7 total)
| File | Context |
|------|---------|
| `main.go` | DM typing indicators |
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
| `handlers/stream_loop.go` | SSE stream events |
| `handlers/tool_loop.go` | Browser tool call delivery |
| `notifications/service.go` | Push notification delivery |
### 4.1 — Add `TargetUserID` to Event
```go
type Event struct {
Label string `json:"event"`
Room string `json:"room,omitempty"`
Payload json.RawMessage `json:"payload"`
Ts int64 `json:"ts"`
// Server-side metadata (not serialized to clients)
SenderID string `json:"-"`
ConnID string `json:"-"`
TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
}
```
`TargetUserID` is included in the JSON serialization for `pg_broadcast`
(so the remote pod knows who to deliver to) but stripped before sending
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
path, or use a separate internal envelope.
**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
harmless if clients see it (they already ignore unknown fields), and
avoids a second serialization path. The field is empty for room-scoped
events and only populated for targeted delivery.
### 4.2 — Update WS Subscriber Filter
In `subscribeToBus`, add target filtering:
```go
// Targeted delivery: if event has a target user, only deliver to that user
if e.TargetUserID != "" && e.TargetUserID != c.userID {
return
}
```
This goes before the room filter. Targeted events skip room filtering
entirely (they're user-scoped, not room-scoped).
### 4.3 — Rewrite `SendToUser` → `PublishToUser`
New method on `Hub`:
```go
// PublishToUser sends an event to a specific user via the bus.
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
func (h *Hub) PublishToUser(userID string, event Event) {
event.TargetUserID = userID
h.bus.Publish(event)
}
```
This replaces all 7 `hub.SendToUser` call sites. The event flows:
1. `hub.PublishToUser(userID, evt)`
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
4. `broadcastHook``pg_notify` → remote pod
5. Remote pod: `publishLocal(evt)``subscribeToBus` filter → deliver
### 4.4 — Keep `SendToUser` as Local-Only Optimization
Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
`PublishToUser` can optionally try local delivery first (fast path)
and only broadcast if the user isn't connected locally. But this
optimization is premature at 50 users — just always go through the
bus. Revisit post-MVP if latency matters.
### 4.5 — Audit: `tool.call` + `WaitFor`
`tool.call.*` events use `DirToClient` routing and are delivered via
`SendToUser`. After the rewrite, they go through the bus and will
broadcast to all pods. The `WaitFor` on the originating pod still
works — `tool.result.*` comes back from the client on the same pod
(the WS connection is sticky to a pod).
Verify: if the user's browser tool is connected to pod-2, but the
completion handler calling `WaitFor` is on pod-1, the tool call event
needs to reach pod-2 (via pg_broadcast), and the result needs to come
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
the broadcast hook skips it. **This needs to change to `DirBoth`** so
tool results cross pod boundaries for `WaitFor` to work.
### Deliverables
- [ ] `TargetUserID` field on `Event`
- [ ] `subscribeToBus` target filter
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
- [ ] Update `tool.result.*` routing to `DirBoth`
- [ ] Rename `SendToUser``sendToUserLocal` (unexported)
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
- [ ] CI green
---
## CS5 — Health Probes + Helm Validation
### 5.1 — Readiness Probe Refinement
Current health endpoint: basic HTTP 200. Add a PG ping:
```go
func (h *HealthHandler) Readiness(c *gin.Context) {
ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
defer cancel()
if err := database.DB.PingContext(ctx); err != nil {
c.JSON(503, gin.H{"error": "database unavailable"})
return
}
c.JSON(200, gin.H{"status": "ok"})
}
```
Wire as `/healthz/ready` (separate from the existing `/healthz/live`
liveness probe). Kubernetes pulls the pod from the service on readiness
failure — new requests route to healthy replicas.
### 5.2 — Helm Changes
`chart/values.yaml`:
```yaml
backend:
replicaCount: 2 # was 1
```
`chart/templates/deployment-backend.yaml` additions:
**Pod anti-affinity** — spread replicas across nodes:
```yaml
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: switchboard-backend
topologyKey: kubernetes.io/hostname
```
`preferredDuring` (not `requiredDuring`) — if fewer nodes than
replicas, pods still schedule on the same node (degraded HA is better
than no scheduling).
**Readiness probe:**
```yaml
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
```
### 5.3 — Startup Jitter
Add random jitter (015s) to the scheduler's initial tick to stagger
replica polling:
```go
func (s *Scheduler) Run() {
// ...
jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
time.Sleep(jitter)
log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
// ...
}
```
Not strictly necessary with `SKIP LOCKED` (contention is harmless),
but reduces unnecessary lock acquisition at startup.
### 5.4 — Validation Checklist
Manual validation on the `gobha-ai-chat` cluster:
- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
- [ ] Create a cron task (every 1 min), confirm only one run per tick
- [ ] WebSocket: connect on pod-1, send message that completes on
pod-2, verify streaming events arrive
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
burst regardless of which replica serves each request
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
continues serving + picks up scheduler duties immediately
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load
### Deliverables
- [ ] `/healthz/ready` endpoint with PG ping
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
- [ ] Scheduler startup jitter
- [ ] Validation checklist executed and documented
- [ ] CI green, tag `v0.32.0`
---
## Summary: Store Interface Additions
```go
type Stores struct {
// ... existing fields ...
Tickets TicketStore // v0.32.0: WS auth tickets (PG-backed)
RateLimits RateLimitStore // v0.32.0: Distributed rate limiting
}
```
`TaskStore` changes:
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`
---
## Changeset Sequence
| CS | Scope | Key Files | Gate |
|----|-------|-----------|------|
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |
---
## Risk Assessment
**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
(rate limiter — auth-only, fail-open).
**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
`--replicas=1`).
**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
handlers, completion, notifications). The `tool.result` routing change
(`DirFromClient``DirBoth`) needs careful testing — if tool results
broadcast to all pods, the non-originating pod's `WaitFor` must not
accidentally consume the result. Verify: `WaitFor` subscribes to
`tool.result.{specific-id}` (exact label match), so only the waiting
goroutine receives it. Safe.
**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
always a safe rollback. Each CS should leave CI green at single-replica.

View File

@@ -0,0 +1,238 @@
# DESIGN-0.33.0 — Observability
Operate the platform without reading Go source code. Six changesets
covering structured logging, Prometheus metrics, OpenAPI docs, Grafana
dashboards, alerting rules, and a built-in admin dashboard.
Depends on: v0.32.0 (Multi-Replica HA, current HEAD).
**Design decision:** Zero new database migrations. All metrics are
in-memory per-pod, scraped by Prometheus. Existing tables
(`provider_health`, `tool_health_windows`, `usage_log`, `audit_log`)
continue serving the admin dashboard via stores.
**Library choices:**
- `log/slog` (Go 1.22 stdlib) — zero-dep structured logging
- `prometheus/client_golang` v1.20+ — canonical Go Prometheus client
- Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
- Swagger UI v5 via CDN — single HTML embed, zero build step
---
## What Already Existed
These required no structural changes, only instrumentation:
- **Health accumulator** — in-memory provider+tool metrics, 60s DB flush
- **`Hub.ConnCount()`** — active WebSocket connection count
- **`database.DB.Stats()`** — Go stdlib DB pool metrics
- **K8s probes** — `/healthz/ready` + `/healthz/live` in Helm chart
- **Middleware skip paths** — `/metrics` already in `SkipPaths` list
- **`google/uuid`** — already in go.mod for request ID generation
---
## CS0 — Structured Logging + Request ID
Foundation changeset. All subsequent work emits structured logs.
### `server/logging/logger.go` (new)
`Init(format, level)` configures the global `slog` logger:
- `format=json``slog.NewJSONHandler(os.Stdout, opts)`
- `format=text``slog.NewTextHandler(os.Stdout, opts)`
- Levels: debug, info, warn, error
### `server/middleware/request_id.go` (new)
Generates UUID per request. Reuses existing `google/uuid`.
Honors inbound `X-Request-Id` header for trace propagation.
Sets `request_id` in Gin context and echoes header on response.
### Middleware chain rewrite
Changed `gin.Default()``gin.New()` with explicit chain:
```
RequestID → Prometheus → Logger → Recovery → CORS
```
Logger middleware rewritten from `log.Printf` to `slog.Info("request", ...)`
with structured fields: method, path, status, latency_ms, client_ip,
request_id, user_id.
### Config additions
- `LogFormat` (env: `LOG_FORMAT`, default: `text`)
- `LogLevel` (env: `LOG_LEVEL`, default: `info`)
---
## CS1 — Prometheus `/metrics` Endpoint
### `server/metrics/metrics.go` (new)
All metric definitions in one file using `promauto`:
| Metric | Type | Labels |
|--------|------|--------|
| `switchboard_http_requests_total` | Counter | method, path_pattern, status |
| `switchboard_http_request_duration_seconds` | Histogram | method, path_pattern |
| `switchboard_websocket_connections` | Gauge | — |
| `switchboard_completion_tokens_total` | Counter | direction, model_id |
| `switchboard_completions_total` | Counter | provider_config_id, model_id, status |
| `switchboard_completion_duration_seconds` | Histogram | provider_config_id, model_id |
| `switchboard_provider_status` | Gauge | provider_config_id |
| `switchboard_db_open_connections` | Gauge | — |
| `switchboard_db_in_use_connections` | Gauge | — |
| `switchboard_db_idle_connections` | Gauge | — |
| `switchboard_db_wait_count` | Gauge | — |
| `switchboard_db_wait_duration_seconds` | Gauge | — |
| `switchboard_task_executions_total` | Counter | status |
| `switchboard_sandbox_executions_total` | Counter | entry_point, status |
### `server/metrics/db_collector.go` (new)
Background goroutine reads `database.DB.Stats()` every 15s, updates
Prometheus gauges.
### `server/middleware/prometheus.go` (new)
Gin middleware. Uses `c.FullPath()` for `path_pattern` label to avoid
cardinality explosion from path parameters (e.g. `/channels/:id`
not `/channels/abc123`).
### Instrumentation points
- `events/ws.go``WebSocketConnections.Inc()/Dec()` on connect/disconnect
- `handlers/completion.go` — token counters in `logUsage()`, completion
duration/status in `recordHealth()`
- `handlers/stream_loop.go` — same instrumentation in `recordHealthFn()`
- `health/accumulator.go``ProviderStatus` gauge updated in `flush()`
### Route
`/metrics` mounted via `promhttp.Handler()` (no auth, standard scraping).
---
## CS2 — OpenAPI Spec + Swagger UI
### `server/static/openapi.yaml` (new)
Hand-curated OpenAPI 3.0.3 spec covering core API groups:
Auth, Channels, Completions, Health, Metrics, WebSocket Tickets,
Admin (stats, health, usage, dashboard).
### `server/static/swagger.html` (new)
Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at
`openapi.yaml`. Zero build step.
### Routes
- `GET /api/docs` → serves embedded `swagger.html`
- `GET /api/docs/openapi.yaml` → serves embedded `openapi.yaml`
Both files embedded via `//go:embed` directives.
---
## CS3 — Grafana Dashboard + Alerting Rules + Helm
Static files + Helm templates. No Go code changes.
### `chart/dashboards/switchboard-overview.json` (new)
Grafana dashboard with 10 panels:
- Request rate, error rate, latency percentiles
- WebSocket connections, completion rate/latency by provider
- Tokens/min by model, provider status
- DB connection pool, task executions
Template variables: `$datasource`, `$namespace`, `$pod`.
### `chart/alerting/switchboard-rules.yaml` (new)
Source PrometheusRule with 6 alerts:
| Alert | Condition | Severity |
|-------|-----------|----------|
| SwitchboardPodRestart | Restarts in 1h | warning |
| SwitchboardProviderDown | Status > 2 for 5m | critical |
| SwitchboardDBPoolExhaustion | In-use/open > 80% for 5m | warning |
| SwitchboardHighErrorRate | 5xx rate > 5% for 5m | warning |
| SwitchboardTaskFailureRate | Error rate > 25% for 10m | warning |
| SwitchboardNoCompletions | Zero completions for 15m | critical |
### Helm templates (new, all gated `enabled: false` by default)
- `chart/templates/servicemonitor.yaml` — ServiceMonitor CRD
- `chart/templates/grafana-dashboard-configmap.yaml` — ConfigMap for
Grafana sidecar discovery
- `chart/templates/prometheusrule.yaml` — PrometheusRule CRD
### Helm modifications
- `chart/values.yaml``logging` section + `monitoring` section
- `chart/templates/configmap.yaml``LOG_FORMAT`, `LOG_LEVEL` env vars
- `chart/templates/services.yaml` — Prometheus scrape annotations
---
## CS4 — Admin Observability Dashboard
Built-in admin page for real-time health without Grafana dependency.
### `server/handlers/dashboard_admin.go` (new)
`GET /api/v1/admin/dashboard` aggregating:
- Provider health summaries (from `HealthStore`)
- 24h usage totals (from `UsageStore`)
- DB pool stats (`database.DB.Stats()`)
- WebSocket connection count (`Hub.ConnCount()`)
- Recent errors (from `AuditStore`)
- Process uptime
### Frontend
- `SCAFFOLDING.dashboard` — stat cards grid + content container
- `ADMIN_LOADERS.dashboard``UI.loadAdminDashboard()`
- `ADMIN_SECTIONS.monitoring``dashboard` added as first section
- Dashboard auto-refreshes every 30s
- CSS: provider grid, DB pool bar gauge, error list
### Admin template
Monitoring tab now lands on `/admin/dashboard` instead of `/admin/usage`.
---
## CS5 — Wiring, docker-compose, ICD Tests
### docker-compose.yml
Added `LOG_FORMAT` and `LOG_LEVEL` env vars with defaults.
### ICD tests (`crud/observability.js`)
| Test | Assertion |
|------|-----------|
| `GET /metrics` | 200, contains `switchboard_*` metrics |
| `GET /api/docs` | 200, contains `swagger-ui` |
| `GET /api/docs/openapi.yaml` | 200, contains `openapi:` and `paths:` |
| X-Request-Id generation | Response includes 36-char UUID header |
| X-Request-Id passthrough | Client-sent header echoed back |
| `GET /admin/dashboard` | 200, has `uptime`, `ws_connections`, `provider_health` |
---
## Verification
1. `LOG_FORMAT=json docker compose up` → JSON log lines on stdout
2. `curl localhost:8080/metrics | grep switchboard_` → all metrics present
3. `curl -I localhost:8080/api/v1/channels``X-Request-Id` header
4. `/api/docs` in browser → Swagger UI renders
5. `/admin` → Monitoring → Dashboard shows live stat cards
6. `helm lint chart/ --set monitoring.serviceMonitor.enabled=true` → passes
7. ICD test suite passes (existing + observability tests)

View File

@@ -0,0 +1,217 @@
# DESIGN-0.34.0 — Data Portability
Export your data, import it elsewhere, delete your account. Six
changesets covering user export, user import, GDPR delete, admin
team export/import, ChatGPT import, and K8s scheduled backups.
Depends on: v0.33.0 (Observability, current HEAD).
**Design decision:** One new migration (021) — indexes only, no new
tables. The export archive is a ZIP file (`.switchboard` extension)
containing a JSON manifest and per-entity JSON files. Sensitive fields
(password hashes, encrypted keys, storage keys) are stripped at export
time via entity sanitization.
**Archive format:** `.switchboard` ZIP with `format_version=1`:
```
manifest.json
data/users.json
data/channels.json
data/messages.json
data/channel_participants.json
data/channel_models.json
data/channel_cursors.json
data/notes.json
data/note_links.json
data/memories.json
data/projects.json
data/project_channels.json
data/project_knowledge_bases.json
data/project_notes.json
data/workspaces.json
data/workspace_files.json
data/folders.json
data/user_settings.json
data/notification_preferences.json
data/usage_entries.json
data/persona_groups.json
data/persona_group_members.json
data/files.json
files/{file_id}/{filename}
```
---
## What Already Existed
- **Package export** — `.pkg` ZIP format, `package_export.go`
- **Markdown export** — pandoc-based, `export.go`
- **Workspace archive** — ZIP/tar.gz with size limits, `workspace/archive.go`
- **Soft deletes** — `deleted_at` on messages
- **Audit trail** — `audit_logs` table
- **39 store interfaces** covering all entity types
---
## CS0 — Export Format + User Export
Foundation changeset. Defines the archive format and ExportStore interface.
### `server/export/format.go` (new)
Archive types: `Manifest`, `ManifestScope`, `ArchiveWriter`, `ArchiveReader`.
Size limits: 500 MB archive, 100 MB per file, 10K files max.
Key methods: `WriteManifest`, `WriteEntityJSON`, `WriteFile`,
`ReadManifest`, `ReadEntityJSON`, `FileEntries`.
### `server/export/entities.go` (new)
Per-entity sanitization functions. Export types strip sensitive fields:
- `ExportChannel` — removes `provider_config_id`
- `ExportMessage` — no changes needed
- `ExportWorkspace` — removes `root_path`, git credentials
- `ExportUsageEntry` — removes `provider_config_id`
- Users — removes `password_hash`, `encrypted_uek`, `uek_salt`
### `server/store/interfaces.go` (modified)
Added `Export ExportStore` to `Stores` struct. The `ExportStore`
interface provides 21 user-scoped read methods, 20 import methods
(returning `(imported, skipped, error)` counts), 14 team-scoped
read methods, and 4 GDPR methods.
### `server/store/{postgres,sqlite}/export.go` (new)
Full implementations for both dialects (~1400-1500 lines each).
Import uses `INSERT ON CONFLICT DO NOTHING` (PG) / `INSERT OR IGNORE`
(SQLite) for UUID-based dedup.
### `server/handlers/export_data.go` (new)
`DataExportHandler.ExportMyData``GET /api/v1/export/me`.
Streams ZIP directly to response using `zip.NewWriter(c.Writer)`.
Includes file blobs from object store.
### Migration 021
Indexes only — `idx_messages_channel_active`, `idx_channels_user_id`,
`idx_files_user_id`. PG version uses partial index
(`WHERE deleted_at IS NULL`).
---
## CS1 — User Import
### `server/handlers/import_data.go` (new)
`DataImportHandler.ImportMyData``POST /api/v1/import/me`.
Opens archive, validates manifest `format_version`, reads entities
in FK-dependency order, remaps `user_id` for cross-instance import.
Returns `{imported: {channels: N, ...}, skipped: {...}, errors: [...]}`.
Import order: Folders → Projects → Workspaces → Channels →
Participants/Models/Cursors → Messages → Notes → Note Links →
Memories → Project links → Workspace files → Files + blobs →
Settings → Persona groups.
---
## CS2 — GDPR Account Delete
### `server/handlers/gdpr.go` (new)
`GDPRHandler.DeleteMyAccount``DELETE /api/v1/me`.
Requires `{"confirm": "DELETE", "password": "..."}`. Verifies
password, prevents last-admin deletion, then:
1. Soft-deletes messages, archives channels
2. Hard-deletes: notes, memories, workspaces, files, projects, settings
3. Revokes tokens (refresh + WS tickets)
4. Anonymizes user: `deleted-user-{sha256(id)[:12]}`
5. Writes audit log entry
---
## CS3 — Admin Team Export/Import
`DataExportHandler.ExportTeam``GET /api/v1/admin/teams/:id/export`
`DataImportHandler.ImportTeam``POST /api/v1/admin/teams/:id/import`
Same archive format, scoped to team entities (channels, personas,
knowledge bases, workflows, groups, projects, resource grants).
Team import skips members whose `user_id` doesn't exist on target.
---
## CS4 — ChatGPT Import
### `server/export/chatgpt.go` (new)
Parses ChatGPT `conversations.json` format. Each conversation maps
to one channel (type=direct). The `mapping` DAG is walked depth-first
to produce ordered messages with `parent_id` and `sibling_index`.
Role mapping: `user`→user, `assistant`→assistant, `system`→system,
`tool`→tool. Content: `parts[]` joined with `\n`, non-string parts
skipped. Timestamps: `float64` epoch → `time.Time`.
`DataImportHandler.ImportChatGPT``POST /api/v1/import/chatgpt`.
Accepts multipart upload of `conversations.json` or ZIP containing it.
---
## CS5 — K8s Backup + ICD Tests
### `chart/templates/cronjob-backup.yaml` (new)
CronJob running `pg_dump | gzip` on schedule (default daily 02:00 UTC).
Optional S3 upload via AWS CLI. Local retention pruning. Gated on
`backup.enabled=true` and `database.driver=postgres`.
### `chart/templates/pvc-backup.yaml` (new)
Conditional PVC for local backup storage (when `backup.persistence.enabled`).
### `chart/values.yaml` (modified)
New `backup:` section with schedule, retention, S3 config, persistence,
and resource limits.
### ICD Tests
~30 new tests in `crud/portability.js` covering:
- Export download + archive validation (ZIP structure, manifest, entities)
- Sensitive field exclusion verification
- Import re-import (dedup validation)
- Invalid archive rejection
- ChatGPT import (valid, invalid JSON, empty array)
- ChatGPT channel/message verification
- GDPR delete flow (register → create data → delete → verify lockout)
- GDPR validation (missing confirm, wrong password)
- Admin team export/import
---
## New Routes
| Method | Path | Auth | Handler |
|--------|------|------|---------|
| GET | `/api/v1/export/me` | JWT | `DataExportHandler.ExportMyData` |
| POST | `/api/v1/import/me` | JWT | `DataImportHandler.ImportMyData` |
| POST | `/api/v1/import/chatgpt` | JWT | `DataImportHandler.ImportChatGPT` |
| DELETE | `/api/v1/me` | JWT | `GDPRHandler.DeleteMyAccount` |
| GET | `/api/v1/admin/teams/:id/export` | Admin | `DataExportHandler.ExportTeam` |
| POST | `/api/v1/admin/teams/:id/import` | Admin | `DataImportHandler.ImportTeam` |
---
## Verification
1. `cd server && go build ./...` — all changesets compile clean
2. `helm template ./chart --set backup.enabled=true` — CronJob renders
3. ICD test suite: ~615 tests (585 existing + ~30 portability)
4. Manual smoke: export → inspect ZIP → re-import → verify dedup
5. GDPR: delete → verify login fails → inspect anonymized DB record
6. Sensitive fields: no `password_hash`, `encrypted_uek`, `storage_key` in export

View File

@@ -0,0 +1,148 @@
# DESIGN-0.35.0 — Workflow Product
Transforms the workflow engine from a linear stage-runner into a
product-grade automation tool. Seven changesets covering conditional
routing, AI-triggered routing, data pipeline hooks, progressive forms,
conditional fields, structured review, and a monitoring dashboard.
Depends on: v0.34.0 (Data Portability), v0.31.2 (Team Workflows),
v0.29.0 (Starlark Sandbox).
## Schema Changes
Migrations modified in-place (DB rebuild required):
**005_channels.sql** — add `stage_entered_at` (PG: `TIMESTAMPTZ`,
SQLite: `TEXT`). Tracks when the current stage was entered for SLA
computation. Also add index `idx_channels_workflow_active` on
`(workflow_id, workflow_status)` for monitoring queries.
**018_workflows.sql** — add `sla_seconds INTEGER` to `workflow_stages`.
Add `review_comments` (PG: `JSONB DEFAULT '[]'`, SQLite: `TEXT
DEFAULT '[]'`) to `workflow_assignments`.
No new migration files. No new tables.
## Conditional Routing Engine
**File:** `server/workflow/routing.go`
The existing advance logic hardcoded `nextStage = currentStage + 1`.
The routing engine replaces this with condition evaluation:
1. Parse `transition_rules.conditions[]` from the departing stage
2. Evaluate each condition against accumulated `stage_data`
3. First match → resolve `target_stage` (by name or ordinal)
4. No match → fallback to `ordinal + 1`
**Expression format:** `{field, op, value, target_stage}`. Operators:
`eq`, `neq`, `gt`, `lt`, `gte`, `lte`, `in`, `contains`, `exists`,
`not_exists`. Loose equality via `fmt.Sprintf("%v")` comparison.
Numeric comparison extracts `float64` from both sides.
**Wiring:** `ResolveNextStage()` called in all 4 advance sites:
handler (`workflow_instances.go`), tool (`tools/workflow.go`), forms
auto-advance (`workflow_forms.go`), Starlark module
(`sandbox/workflow_module.go`).
**Backward compatible:** no conditions = `ordinal + 1`.
## AI-Triggered Routing
**Tool:** `workflow_route` — persona calls with `{target_stage, reason}`.
Uses `ResolveStageByName()` (case-insensitive). Records decision in
`stage_data._route_history_latest`. Allows forward and backward jumps
(loop stages for iterative correction).
**Starlark:** `workflow.route(channel_id, target_stage, reason)` for
extension-driven escalation patterns. Requires `workflow.access`.
## on_advance Hook
**File:** `server/handlers/workflow_hooks.go`
Fires synchronously after `AdvanceWorkflowStage` succeeds. Configured
per-stage in `transition_rules.on_advance: {package_id, entry_point}`.
Hook receives: `{stage_data, previous_stage, current_stage, channel_id}`.
Returns: `{stage_data: {...}}` (enriched), `None` (no change), or
`{error: "msg"}` (logged, not rolled back — future: rollback support).
The existing `http` Starlark module (v0.29.1) is available in hooks,
enabling external API enrichment without new infrastructure.
## Progressive Forms
**Model:** `TypedFormTemplate.Fieldsets []FormFieldset` — optional
array of `{label, fields[]}`. When present, top-level `fields` is
replaced by flattened fieldset fields in `ParseTypedFormTemplate()`.
**Frontend:** `workflow.html` renders one fieldset at a time with
step indicator, next/back navigation. All fields submitted together
on the final step.
## Conditional Fields
**Model:** `FormField.Condition *FieldCondition``{when, op, value}`.
Operators: `eq` (default), `neq`, `in`, `exists`.
**Server-side:** `ValidateFormData()` calls `evaluateFieldCondition()`
before each field. Hidden fields are skipped (no required check, no
type validation).
**Client-side:** `workflow.html` attaches `change`/`input` listeners
to source fields. Target fields toggle `cond-hidden` CSS class.
## Structured Review
**Review comments:** `review_comments JSONB` on `workflow_assignments`.
`POST /workflow-assignments/:id/comment` appends `{text, user_id,
created_at}`. `GET /workflow-assignments/:id` returns full assignment.
**Review surface:** Side-by-side layout in `workflow.html` — left panel
shows structured `stage_data` table (internal `_`-prefixed keys hidden),
right panel has comment textarea + approve/reject buttons. Keyboard
shortcuts: `Ctrl+Enter` approve, `Ctrl+Shift+Enter` reject.
## Monitoring Dashboard
**File:** `server/handlers/workflow_monitor.go`
**Endpoints:**
- `GET /admin/workflows/monitor/instances` — all active instances
- `GET /admin/workflows/monitor/funnel/:id` — stage counts
- `GET /admin/workflows/monitor/stale?threshold_hours=N`
- `GET /teams/:teamId/workflows/monitor/instances` — team-scoped
**SLA computation:** `stage_entered_at + sla_seconds` vs `NOW()`.
Returned as `sla_remaining_seconds` (negative = breached) and
`sla_breached` boolean. Computed at query time, not stored.
**Store:** New `ListByType(ctx, "workflow")` method on `ChannelStore`
(both PG and SQLite). Monitoring queries iterate active channels,
joining workflow definitions and stage metadata.
**Frontend:** `src/js/workflow-monitor.js` — auto-refresh every 30s,
SLA color indicators (green > 50%, yellow > 20%, red), stale instance
alert section.
## Workflow Branding
`WorkflowPageData.BrandingJSON` passes the workflow's `branding` JSON
to the visitor template. Applied as:
- `--accent` CSS custom property (from `accent_color`)
- Logo image in header (from `logo_url`)
- Tagline text below header (from `tagline`)
Previously branding was only on the landing page; now also on the
active workflow surface.
## New Files
| File | Purpose |
|------|---------|
| `server/workflow/routing.go` | Conditional routing engine |
| `server/handlers/workflow_hooks.go` | on_advance hook dispatch |
| `server/handlers/workflow_monitor.go` | Monitoring dashboard |
| `src/js/workflow-monitor.js` | Admin monitoring frontend |
| `packages/icd-test-runner/js/crud/workflow-product.js` | E2E tests |

View File

@@ -0,0 +1,773 @@
# DESIGN — Surface & Extension Architecture
**Status:** Accepted (Phases 12 implemented in v0.25.1)
**Scope:** Primitives, Components, Surfaces, Extension hooks, Themes
**Depends on:** v0.22.5 (Go template engine), v0.22.7 (ChatPane), v0.25.1 (audit)
**Informs:** v0.22.8+, EXTENSIONS.md rewrite, ARCHITECTURE.md update, ICD-API, ICD-SURFACE
**Note:** For current package/surface authoring, see [PACKAGES.md](PACKAGES.md) and [EXTENSION-SURFACES.md](EXTENSION-SURFACES.md). For workflow surfaces, see [WORKFLOW-PACKAGES.md](WORKFLOW-PACKAGES.md).
---
## Overview
The UI is a four-layer architecture: Primitives, Components, Surfaces,
and Banners. Each layer has a single responsibility and a strict
dependency direction — primitives know nothing, components compose
primitives, surfaces compose components, banners are platform chrome
outside the surface boundary.
Extensions participate at every layer through six defined hooks.
Admin-installed extensions are privileged (full DOM, direct API access,
own routes). Themes are pure CSS custom property overrides that
propagate through every layer uniformly.
This document defines the layer contracts, extension hook system,
display content model, trust boundaries, and theme architecture.
---
## Definitions
A **surface** is a full-page layout — a composition of components with
a route, a data loader, and a boot script. Chat is a surface. The
editor is a surface. A custom triage intake form is a surface.
An **extension** is a package — a manifest plus code plus assets that
participates in the platform through hooks. An extension can do one,
some, or all of:
- Register tools the LLM can call (Hook 5: Tool Bridge)
- Transform content during streaming (Hook 2: Stream Processing)
- Enhance rendered messages (Hook 3: Post-Render)
- Attach visual content to messages (Hook 4: Display Content)
- Inject panels or sections into existing surfaces (Hook 6: Surface Injection)
- Create entirely new surfaces with their own routes (Surface Registration)
The five **core surfaces** (Chat, Editor, Notes, Admin, Settings) exist
without extensions. They are built from the same primitive and component
layers that extensions use, but they are registered directly in Go
rather than through a manifest.
An extension that creates a surface gets its own route (`/s/:slug`),
its own data loader, and the full primitive/component library. An
extension that only registers a post-render hook (like a block
renderer) doesn't create any surfaces at all.
**Examples:**
| Extension | Creates Surface? | Hooks Used |
|-----------|-----------------|------------|
| Mermaid renderer | No | Post-Render |
| KaTeX renderer | No | Post-Render |
| Calculator tool | No | Tool Bridge |
| Image Generator | Yes (`/s/gallery`) | Tool Bridge, Stream Processing, Post-Render, Display Content, Surface |
| Custom Dashboard | Yes (`/s/dashboard`) | Surface, Surface Injection (admin section) |
---
## Existing Extension Mapping
The current extension mechanisms map directly into this architecture:
| Current Mechanism | Architecture Equivalent |
|-------------------|------------------------|
| Block renderers (`ctx.renderers.register()`) | Hook 3: Post-Render |
| Tool bridge (`ctx.tools.register()`) | Hook 5: Tool Bridge |
| `ctx.ui.toast()`, `ctx.ui.openPreview()` | Primitive layer (unchanged API) |
| `ctx.ui.isDark()`, `ctx.ui.isMobile()` | Theme layer queries |
| `ctx.surfaces.getCurrent()` | Returns `window.__SURFACE__` |
The manifest schema gains optional new fields (`hooks`, `surfaces`,
`surface_injections`, `theme`). Existing fields retain their meaning.
`Extensions.boot()` is the single entry point for extension
initialization on every surface. It loads manifests, registers
renderers, and initializes tool bridges — same sequence, available
on every surface rather than only chat.
---
## Layer Model
```
┌──────────────────────────────────────────────┐
│ Banner (top) │ ← Platform config
├──────────────────────────────────────────────┤
│ │
│ Surface │ ← Layout + lifecycle
│ ┌─────────────┬──────────────────────────┐ │
│ │ Component │ Component │ │ ← Domain-aware
│ │ (ChatPane) │ (NoteEditor) │ │
│ │ ┌─────────┐ │ ┌────────┐ ┌──────────┐ │ │
│ │ │Primitive│ │ │Primitv.│ │Primitive │ │ │ ← Atomic UI
│ │ │ (input) │ │ │ (menu) │ │ (toggle) │ │ │
│ │ └─────────┘ │ └────────┘ └──────────┘ │ │
│ └─────────────┴──────────────────────────┘ │
│ │
├──────────────────────────────────────────────┤
│ Banner (bottom) │ ← Platform config
└──────────────────────────────────────────────┘
```
Four layers, strict dependency direction: Primitives know nothing.
Components use Primitives. Surfaces compose Components. Banners are
global chrome outside the surface boundary.
---
## Layer 1: Primitives
Atomic UI elements. No domain logic. A toggle doesn't know if it's
toggling a KB or a theme — it takes a label, a state, and a callback.
Styled entirely via CSS custom properties (the theme contract). Every
primitive reads from the same property namespace, so theme changes
propagate instantly.
### Catalog
| Primitive | Description |
|-----------|-------------|
| `Input` | Text, textarea, number, password, with label + validation |
| `Select` | Dropdown, single or multi |
| `Toggle` | Boolean switch with label |
| `Checkbox` | Checkbox with label |
| `Button` | Text button, variants: primary, secondary, danger, ghost |
| `IconButton` | Icon-only button with tooltip |
| `ColorPicker` | Color input with hex text field |
| `Menu` | Dropdown or context menu, item list with icons + shortcuts |
| `Dialog` | Modal: confirm, prompt, or custom form content |
| `Toast` | Transient notification: success, error, warning, info |
| `Badge` | Inline label: accent, success, danger, warning, muted |
| `Avatar` | Image circle with upload affordance |
| `Tabs` | Tab bar with content switching |
| `FormGroup` | Label + input + validation message + help text |
| `SectionHeader` | Titled section divider |
| `EmptyState` | Placeholder with icon + message + action |
| `Spinner` | Loading indicator |
| `Table` | Sortable, paginated data table |
### Implementation
Each primitive is a factory function that returns a DOM element (or
attaches to an existing one). No classes, no inheritance — just
functions.
```js
// Example — not prescriptive API, just the pattern:
Primitives.toggle({ label: 'Auto-search', value: true, onChange: fn })
Primitives.menu({ anchor: el, items: [...], onSelect: fn })
Primitives.dialog({ title: 'Confirm', body: el, onConfirm: fn })
```
Current locations: `ui-primitives.js` (factory functions, `esc()`,
`createComponentRegistry()`, `componentMixin()`, `showConfirm()`,
`showPrompt()`, `Theme`, `mountAvatarUpload()`)
and Go template components (`model-select.html`, `team-select.html`,
`file-upload.html`). These converge into one coherent set.
### Go Template Primitives
Some primitives need server-rendered initial state (e.g., model-select
needs the model list, team-select needs the team list). These are Go
template partials that render the HTML + initial data, then JS hydrates
interactivity on load.
```html
{{template "model-select" dict "ID" "channelModel" "Models" .Models "Type" "chat"}}
```
The JS primitive attaches to the server-rendered DOM by ID, not by
replacing it.
---
## Layer 2: Components
Domain-aware compositions of primitives. Each component owns its DOM
subtree, manages its own state, and exposes a clean API for the surface
to interact with.
### Catalog
| Component | Primitives Used | Domain |
|-----------|----------------|--------|
| `ChatPane` | Input, Button, Menu, Spinner, Toast | Channels, Messages, Streaming |
| `NoteEditor` | Input (CM6), Menu, Tabs, Badge | Notes, Wikilinks, Folders |
| `FileTree` | Menu (context), Button, Spinner | Workspaces, Files |
| `ModelSelector` | Select, Badge, Spinner | Models, Capabilities, Health |
| `PersonaPicker` | Select, Badge, Avatar | Personas, Scopes |
| `KBPicker` | Toggle, Badge, Select | Knowledge Bases, Discoverability |
| `ProjectSidebar` | Menu, Tabs, Badge, EmptyState | Projects, Channels, DnD |
| `AuditLog` | Table, Select, Badge | Audit, Filtering, Pagination |
| `CodeEditor` | Input (CM6), Select, Tabs | Workspaces, Languages |
| `SettingsForm` | FormGroup, Toggle, Select, Button | User/Admin Settings |
### Instance Pattern
Components use `createComponentRegistry()` and `componentMixin()` from
`ui-primitives.js` for shared lifecycle (instance tracking, event
listener cleanup, destroy):
```js
const ChatPane = {
...createComponentRegistry('ChatPane'),
create(opts) {
const instance = componentMixin({
id: opts.id || 'pane-' + Date.now(),
messagesEl: opts.messagesEl,
inputEl: opts.inputEl,
channelId: opts.channelId || null,
// ... domain-specific state
// Component-specific teardown (called by mixin's destroy())
_cleanup() {
if (this._cmView) { this._cmView.destroy(); this._cmView = null; }
},
}, ChatPane);
ChatPane._register(instance.id, instance);
return instance;
},
// Custom queries beyond .get(id)
forChannel(channelId) { /* ... */ },
};
```
The mixin provides `_on(el, event, handler)` for tracked event binding
and `destroy()` which calls `_cleanup()` then removes all listeners and
unregisters from the registry. Components that need extra teardown define
`_cleanup()`.
Components can be instantiated multiple times on the same page (editor
has a ChatPane, notes has a ChatPane — independent instances).
### Server-Rendered Shell + JS Hydration
Components have a Go template partial for the server-rendered scaffold:
```html
{{template "chat-pane" dict "ID" "editor"}}
```
This renders the DOM structure with predictable IDs. The JS component
attaches to these IDs on DOMContentLoaded. No client-side DOM
construction for the initial layout.
---
## Layer 3: Surfaces
A surface is a full-page layout that composes components. Each surface
declares:
1. **What components it uses** (ChatPane, NoteEditor, FileTree, etc.)
2. **How they're arranged** (CSS grid/flex layout)
3. **What data it needs on load** (Go data loader)
4. **What JS runs on boot** (script block or module)
### Current Surfaces
| Surface | Components | Data Loader |
|---------|-----------|-------------|
| Chat | ChatPane, ProjectSidebar, ModelSelector, PersonaPicker | channels, personas, models, projects |
| Editor | FileTree, CodeEditor, ChatPane (assist) | workspace, files, models |
| Notes | NoteEditor, NoteGraph, ChatPane (assist) | notes, folders, graph |
| Admin | AuditLog, SettingsForm, Table (various) | users, configs, models, health |
| Settings | SettingsForm, KBPicker, PersonaPicker | profile, preferences, policies |
### Surface Registration
Today: hardcoded in Go (`pageEngine.RenderSurface("chat")`).
Future: manifest-driven. An admin-installed extension declares a
surface in its manifest:
```json
{
"surfaces": [{
"id": "triage-intake",
"route": "/s/triage",
"title": "Triage Intake",
"components": ["chat-pane", "settings-form"],
"data_requires": ["personas", "models"],
"script": "surfaces/triage.js",
"auth": "authenticated"
}]
}
```
The page engine reads registered surfaces, generates routes, assembles
data loaders from the declared requirements, and renders a shell
template that loads the surface's script.
**Route namespace:** Admin-created surfaces live under `/s/:slug` to
avoid collision with core routes.
### Data Loaders
Each surface declares what data it needs. The page engine has a
registry of data providers:
```
"personas" → loads personas for the current user
"models" → loads enabled models with health status
"channels" → loads user's channels
"workspace" → loads workspace by :wsId param
"notes" → loads notes list
...
```
The surface's `data_requires` field pulls from this registry. Data is
injected into the page as `window.__PAGE_DATA__` (existing pattern).
The surface's JS reads from there on boot — no waterfall of API calls
on page load.
---
## Layer 4: Banners
The banner is global chrome outside the surface boundary. It exists
at the top, bottom, or both — configured by the platform admin via
`PUT /admin/settings/banner`.
```json
{
"enabled": true,
"text": "DEVELOPMENT",
"position": "both|top|bottom",
"bg": "#007a33",
"fg": "#ffffff"
}
```
The Go template base layout renders banners before and after the
surface content area. CSS custom properties (`--banner-top-height`,
`--banner-bottom-height`, `--banner-bg`, `--banner-fg`) allow surfaces
to account for banner space without knowing banner state.
Banners are not configurable by extensions or themes. They are a
platform-level trust signal.
---
## Themes
A theme is a set of CSS custom property overrides applied to `<html>`.
```css
[data-theme="corporate"] {
--bg: #f5f5f5;
--text: #1a1a1a;
--accent: #0066cc;
--border: #d1d5db;
--font-ui: 'Inter', sans-serif;
--font-code: 'Fira Code', monospace;
--radius: 4px;
/* ... full property set */
}
```
### What a Theme Can Do
- Override any CSS custom property in the theme contract
- Change colors, fonts, border radii, spacing scale
- Switch between light and dark base palettes
- Apply to every primitive, component, and surface uniformly
### What a Theme Cannot Do
- Add or remove DOM elements
- Execute JavaScript
- Modify component behavior or layout
- Override banner appearance (platform chrome)
- Access APIs or user data
### Theme Contract
The set of CSS custom properties that all primitives read from. This
is the stable API between themes and the UI. Properties are namespaced:
```
--bg, --bg-secondary, --bg-tertiary (backgrounds)
--text, --text-secondary, --text-muted (typography)
--accent, --accent-hover, --accent-muted (interactive)
--border, --border-strong (edges)
--success, --warning, --danger (semantic)
--font-ui, --font-code (typefaces)
--radius, --radius-lg (shapes)
--shadow, --shadow-lg (elevation)
--banner-bg, --banner-fg (read-only, set by platform)
```
Themes are admin-installed. An admin uploads a CSS file that declares
a `[data-theme="name"]` rule set. Users can select from installed
themes in Settings → Appearance.
Built-in themes: `dark` (default), `light`. The system preference
auto-detection (`prefers-color-scheme`) continues to work.
---
## Extension Hooks
Extensions interact with the application through a defined set of
hooks. Each hook has a specific trigger point in the lifecycle and
a clear contract for what the extension receives and returns.
### Hook 1: Pre-Completion
**When:** After the user sends a message, before the API request fires.
**Receives:** The completion request object (model, messages, tools, etc.)
**Returns:** Modified request object (or unmodified to pass through).
**Use case:** Inject additional tools, modify system prompt, add context.
```js
ctx.hooks.preCompletion(request => {
request.tools.push(myCustomTool);
return request;
});
```
### Hook 2: Stream Processing
**When:** On each SSE chunk during streaming.
**Receives:** The chunk (content delta, tool_use, tool_result, etc.)
**Returns:** Modified chunk (or null to suppress).
**Use case:** Transform content, intercept tool calls, accumulate data.
```js
ctx.hooks.streamChunk((chunk, context) => {
if (chunk.type === 'tool_result' && chunk.name === 'image_gen') {
context.displayContent.push({ type: 'image', src: chunk.data.url });
return null; // suppress from text content
}
return chunk;
});
```
### Hook 3: Post-Render
**When:** After a message is rendered into the DOM.
**Receives:** The message container element, the message object.
**Returns:** Nothing (mutates DOM in place).
**Use case:** Add action buttons, wrap elements, enhance display.
This is how extensions compose with each other's output. Extension A
produces an image via display content. Extension B's post-render hook
finds `<img>` elements and wraps them with action buttons. They don't
know about each other — they agree on the DOM contract.
```js
ctx.hooks.postRender((containerEl, message) => {
containerEl.querySelectorAll('img[data-display-content]').forEach(img => {
const actions = document.createElement('div');
actions.className = 'image-actions';
actions.innerHTML = '<button data-action="upscale">Upscale</button>';
img.parentElement.appendChild(actions);
});
});
```
### Hook 4: Display Content
**When:** After a completion finishes (all chunks received).
**Receives:** The assistant message object, accumulated display content.
**Returns:** Display content items to attach to the message.
Display content is **message-scoped, rendered inline, but excluded from
the LLM context window**. It's not in `messages.content` and not sent
back on the next turn.
```
Message
├── content (text — goes to LLM)
├── attachments (files — go to LLM via multimodal assembly)
└── display_content[] ← rendered inline, NOT sent to LLM
├── { type: "image", src: "...", extension_id: "img-gen" }
└── { type: "html", content: "<div>...", extension_id: "editor" }
```
**Storage:** Display content is persisted in a `display_content` JSONB
column on the message (or a junction table). It survives page reload
and is included in `GET /channels/:id/path` responses.
### Hook 5: Tool Bridge
**When:** The LLM invokes a tool registered by the extension.
**Receives:** Tool name and input parameters.
**Returns:** Tool result (string or structured).
Existing mechanism — the WebSocket tool bridge from EXTENSIONS.md.
Tool is registered via manifest, exposed to the LLM through the
tools list, and executed client-side (browser tier) or server-side
(starlark/sidecar tiers).
### Hook 6: Surface Injection
**When:** Surface boot (DOMContentLoaded).
**Receives:** The surface ID and available mount points.
**Returns:** Nothing (attaches to mount points).
Extensions declare which surfaces they target and which mount points
they use:
```json
{
"surface_injections": [{
"surface": "chat",
"mount_point": "side-panel",
"component": "my-panel.js"
}, {
"surface": "admin",
"mount_point": "section",
"section_id": "my-admin-section",
"label": "Image Gen Settings"
}]
}
```
Mount points are declared by each surface template:
```html
<div data-mount="side-panel"></div>
<div data-mount="section" data-section-id="..."></div>
```
---
## Trust Model
Two tiers. The admin is the trust boundary.
### Privileged (Admin-Installed)
| Capability | Allowed |
|-----------|---------|
| Full DOM access | Yes |
| Direct API calls (user's JWT) | Yes |
| Modify other extensions' output (post-render) | Yes |
| Create surfaces (own routes) | Yes |
| Register tools | Yes |
| Access all hook types | Yes |
| Go template data loader | Yes |
| Read `window.__PAGE_DATA__` | Yes |
The admin chose to install it. Same trust model as a VS Code extension
or a WordPress plugin — you trust the publisher.
### Sandboxed (User-Installed, Future)
| Capability | Allowed |
|-----------|---------|
| Shadow DOM / iframe only | Yes |
| API calls through message bridge | Yes (proxied, scoped) |
| Modify other extensions' output | No |
| Create surfaces | No |
| Register tools | Limited (user-scoped) |
| Access hooks | Post-render own output only |
| Go template data loader | No |
| Read `window.__PAGE_DATA__` | No |
Post-1.0 scope. Documented here to confirm the architecture
accommodates it without redesign.
---
## Extension Lifecycle
### Boot Sequence
```
1. Page loads (Go template renders surface shell + banner)
2. Primitives initialize (CSS loaded, JS factories available)
3. Components hydrate (attach to server-rendered DOM)
4. Extensions.boot() — idempotent, runs on every surface
a. Load extension manifests
b. Register hooks (pre-completion, post-render, etc.)
c. Register surface injections for current surface
d. Initialize tool bridges
5. Surface-specific JS runs (app.js for chat, editor-surface.js, etc.)
```
`Extensions.boot()` replaces the current chat-specific
`Extensions.loadAll()` / `Extensions.initAll()` pair and runs on
every surface.
### Manifest Declaration
```json
{
"id": "image-generator",
"name": "AI Image Generator",
"version": "1.0.0",
"tier": "browser",
"hooks": {
"pre_completion": "hooks/pre-completion.js",
"stream_chunk": "hooks/stream.js",
"post_render": "hooks/post-render.js"
},
"tools": [{
"name": "generate_image",
"description": "Generate an image from a text description",
"parameters": { ... }
}],
"surfaces": [{
"id": "image-gallery",
"route": "/s/gallery",
"title": "Image Gallery",
"components": ["chat-pane"],
"data_requires": ["channels"],
"script": "surfaces/gallery.js"
}],
"surface_injections": [{
"surface": "chat",
"mount_point": "message-actions",
"script": "injections/image-actions.js"
}],
"theme": null,
"settings_schema": { ... }
}
```
---
## Display Content — Data Model
### Message Extension
```sql
ALTER TABLE messages ADD COLUMN display_content JSONB DEFAULT '[]';
```
Array of display items:
```json
[
{
"type": "image",
"src": "data:image/png;base64,...",
"alt": "A cat in a spacesuit",
"extension_id": "image-generator",
"metadata": { "model": "dall-e-3", "size": "1024x1024" }
},
{
"type": "html",
"content": "<div class='chart'>...</div>",
"extension_id": "data-viz",
"metadata": {}
}
]
```
### API Contract
Display content is included in message objects returned by
`GET /channels/:id/path` and `GET /channels/:id/messages`.
It is **excluded** from the message array sent to the LLM provider
in `POST /chat/completions`. The completion handler strips
`display_content` when assembling the provider request.
### Rendering
During `renderMessages()`, after the message content is rendered,
each `display_content` item is rendered below the text content:
```html
<div class="message-content">
<p>Here's the image you requested:</p>
</div>
<div class="message-display-content">
<img src="..." alt="..." data-display-content data-extension="image-generator">
</div>
```
The `data-display-content` attribute is the DOM contract that
post-render hooks use to discover display content from any extension.
---
## Migration Path
| Current State | Target State |
|---------------|-------------|
| `ui-primitives.js` (merged, with `esc()`, `createComponentRegistry()`, `componentMixin()`) | ✅ Done (v0.25.1) |
| `ChatPane.create()` + 5 other components using shared mixin | ✅ Done (v0.25.1) |
| 5 Go template routes | Surface registry (hardcoded core, manifest-driven extensions) |
| `ctx.ui.replace()` / `ctx.ui.inject()` | Hook 6: Surface Injection with declared mount points |
| `ctx.surfaces.register()` | Manifest `surfaces` field → page engine route registration |
| Block renderers in `Extensions.initAll()` | `Extensions.boot()` on every surface |
| Message content only (text + attachments) | `display_content` column for render-only visual content |
| Block renderer pipeline (mermaid, katex) | Post-render hook + DOM contract (`data-display-content`) |
| `[data-theme]` with CSS variables | Theme contract: stable CSS custom property namespace |
---
## Implementation Sequence
This does not prescribe version numbers. It describes dependency order.
**Phase 1: Primitive consolidation** ✅ (v0.25.1)
- `ui-primitives-additions.js` deleted, survivors merged into
`ui-primitives.js`
- `esc()` centralized (13 private copies eliminated)
- Badge system unified (24 scattered classes → grouped in primitives.css)
- CSS variable names normalized, duplicate selectors resolved
- `showConfirm()` is the sole confirm pattern (bare `confirm()` purged)
**Phase 2: Component formalization** ✅ (v0.25.1)
- `createComponentRegistry()` + `componentMixin()` extracted
- 6 components (ChatPane, ModelSelector, UserMenu, CodeEditor, FileTree,
NoteEditor) use the shared mixin. ~100 lines of boilerplate eliminated.
- Each component has a Go template partial + JS hydration
- `surface-nav.js` deleted; navigation is native in `ui-core.js`
- Vendor scripts (`marked.min.js`, `purify.min.js`) and `ui-format.js`
moved to `base.html` (loaded once, available to all surfaces)
**Phase 3: Extension hooks**
- `Extensions.boot()` on every surface (v0.22.8)
- Pre-completion and post-render hooks
- Display content data model (message column + render pipeline)
- Stream processing hook
**Phase 4: Surface registry**
- Manifest-driven surface registration
- Dynamic route generation in page engine
- Data loader registry
- Mount point declaration and injection
**Phase 5: Theme system**
- Stable CSS custom property contract
- Admin theme upload
- User theme selection in Settings → Appearance
---
## Open Questions
1. **Display content storage:** JSONB column on messages vs. separate
`message_display_content` table. Column is simpler; table allows
independent lifecycle (delete display content without touching
message). Leaning column for KISS.
2. **Display content size limits:** Base64 images in JSONB could get
large. May need blob storage for display content with URL references
instead of inline data. Deferred until real usage patterns emerge.
3. **Extension ordering:** Post-render hooks from multiple extensions
run in what order? Manifest priority field? Installation order?
Alphabetical? Matters when Extension B adds buttons to Extension A's
images — A must render first.
4. **Surface layout persistence:** If a surface supports resizable
panes (editor: file tree width, chat pane width), where is that
persisted? User settings? Per-workspace? Per-surface?
5. **Extension settings storage:** Extensions need per-user config
(API keys for image gen, preferences for behavior). Current
`UpdateUserExtensionSettings` endpoint exists. Is the schema
sufficient or does it need structured validation from
`settings_schema` in the manifest?