Changeset 0.36.0 (#212)
This commit is contained in:
810
docs/archive/DESIGN-0.32.0.md
Normal file
810
docs/archive/DESIGN-0.32.0.md
Normal file
@@ -0,0 +1,810 @@
|
||||
# DESIGN-0.32.0 — Multi-Replica HA
|
||||
|
||||
Run 2–3 backend replicas across nodes for node-level availability.
|
||||
Five changesets. No sub-versions — the work is tightly coupled and
|
||||
each CS builds on the previous.
|
||||
|
||||
Depends on: v0.31.2 (current HEAD).
|
||||
|
||||
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
|
||||
leader election for the task scheduler. Every replica polls; PG
|
||||
serializes the claims. Extends to all shared mutable state — tickets,
|
||||
rate counters — keeping PG as the sole coordination layer. No Redis,
|
||||
no K8s API dependency, no new infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## What Already Works Multi-Replica
|
||||
|
||||
These require zero changes:
|
||||
|
||||
- **REST API** — stateless, JWT auth, any replica serves any request.
|
||||
- **PG + S3 + CephFS** — shared storage infrastructure.
|
||||
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
|
||||
→ `broadcastHook` → `pg_notify` → remote pod `publishLocal()` →
|
||||
local WS subscribers. Fully wired, just never tested at N>1.
|
||||
|
||||
---
|
||||
|
||||
## What Needs Work
|
||||
|
||||
Five areas of in-process mutable state that break at replica count > 1:
|
||||
|
||||
| State | Current | Problem at N>1 | Fix |
|
||||
|-------|---------|----------------|-----|
|
||||
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
|
||||
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
|
||||
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
|
||||
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
|
||||
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Schema: `020_ha.sql`
|
||||
|
||||
New migration for both PG and SQLite. No changes to existing 019
|
||||
task schema — `next_run_at` nullable already supports the claim
|
||||
mechanism.
|
||||
|
||||
### PG: `server/database/migrations/020_ha.sql`
|
||||
|
||||
```sql
|
||||
-- ==========================================
|
||||
-- Chat Switchboard — 020 Multi-Replica HA
|
||||
-- ==========================================
|
||||
-- Shared state tables for multi-replica operation.
|
||||
-- v0.32.0: ws_tickets, rate_limit_counters.
|
||||
-- ==========================================
|
||||
|
||||
-- =========================================
|
||||
-- WEBSOCKET TICKETS (replaces sync.Map)
|
||||
-- =========================================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||||
id TEXT PRIMARY KEY, -- 128-bit hex token
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||||
ON ws_tickets (expires_at);
|
||||
|
||||
-- =========================================
|
||||
-- RATE LIMIT COUNTERS (replaces in-memory)
|
||||
-- =========================================
|
||||
-- Token bucket approximation using time-windowed counters.
|
||||
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
|
||||
-- Window is truncated to the second for the configured rate.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||||
key TEXT NOT NULL,
|
||||
window TIMESTAMPTZ NOT NULL, -- truncated timestamp (window start)
|
||||
tokens REAL NOT NULL DEFAULT 0, -- tokens consumed in this window
|
||||
PRIMARY KEY (key, window)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||||
ON rate_limit_counters (window);
|
||||
```
|
||||
|
||||
### SQLite: `server/database/migrations/sqlite/020_ha.sql`
|
||||
|
||||
SQLite parity — structurally identical but with dialect adjustments.
|
||||
These tables are functional in SQLite for single-process test parity,
|
||||
though multi-replica is PG-only in production.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||||
ON ws_tickets (expires_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||||
key TEXT NOT NULL,
|
||||
window TEXT NOT NULL,
|
||||
tokens REAL NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (key, window)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||||
ON rate_limit_counters (window);
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [x] `020_ha.sql` (PG)
|
||||
- [x] `020_ha.sql` (SQLite)
|
||||
- [x] CI green on both pipelines (migration runs, tables exist)
|
||||
|
||||
---
|
||||
|
||||
## CS1 — Task Scheduler: SKIP LOCKED
|
||||
|
||||
Replace the read-then-execute scheduler with atomic claim semantics.
|
||||
Every replica runs the poll loop. PG serializes task handoff.
|
||||
|
||||
### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)
|
||||
|
||||
New store method. Single atomic statement — SELECT + UPDATE in one
|
||||
round trip. Returns at most one task per call.
|
||||
|
||||
**PG implementation:**
|
||||
|
||||
```sql
|
||||
UPDATE tasks
|
||||
SET next_run_at = NULL
|
||||
WHERE id = (
|
||||
SELECT id FROM tasks
|
||||
WHERE is_active = true
|
||||
AND next_run_at <= NOW()
|
||||
ORDER BY next_run_at ASC
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING <taskColumns>
|
||||
```
|
||||
|
||||
Mechanics:
|
||||
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
|
||||
candidate row, skip it instantly (no wait, no deadlock).
|
||||
- `SET next_run_at = NULL` — claimed task disappears from future
|
||||
polls. `advanceNextRun` restores it after execution.
|
||||
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
|
||||
- The existing partial index `idx_tasks_next_run` covers the WHERE
|
||||
clause (`is_active = true AND next_run_at IS NOT NULL` by
|
||||
implication — NULL rows won't satisfy `<= NOW()`).
|
||||
|
||||
**SQLite implementation:**
|
||||
|
||||
SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
|
||||
the current `ListDue` behavior unchanged for SQLite. The store
|
||||
interface accommodates both:
|
||||
|
||||
```go
|
||||
// TaskStore additions
|
||||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||||
```
|
||||
|
||||
SQLite `ClaimDueTask` just does:
|
||||
```sql
|
||||
SELECT <taskColumns> FROM tasks
|
||||
WHERE is_active = 1 AND next_run_at <= datetime('now')
|
||||
ORDER BY next_run_at ASC LIMIT 1
|
||||
```
|
||||
Then sets `next_run_at = NULL` in a second statement (single writer,
|
||||
no contention).
|
||||
|
||||
### 1.2 — Conditional `CreateRunExclusive`
|
||||
|
||||
Belt-and-suspenders: prevent double-execution even if two replicas
|
||||
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
|
||||
but defense in depth).
|
||||
|
||||
```sql
|
||||
INSERT INTO task_runs (task_id, status)
|
||||
SELECT $1, 'running'
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM task_runs
|
||||
WHERE task_id = $1 AND status IN ('running', 'queued')
|
||||
)
|
||||
RETURNING id, started_at
|
||||
```
|
||||
|
||||
Returns `sql.ErrNoRows` if a run already exists → skip execution.
|
||||
|
||||
New store method:
|
||||
```go
|
||||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
```
|
||||
|
||||
### 1.3 — Scheduler Loop Rewrite
|
||||
|
||||
`scheduler.go` changes:
|
||||
|
||||
```go
|
||||
func (s *Scheduler) poll() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
|
||||
if !cfg.Enabled {
|
||||
return
|
||||
}
|
||||
|
||||
// Claim tasks one at a time until none remain or budget exhausted.
|
||||
// Each replica claims independently — PG SKIP LOCKED serializes.
|
||||
claimed := 0
|
||||
for claimed < cfg.MaxConcurrent {
|
||||
task, err := s.stores.Tasks.ClaimDueTask(ctx)
|
||||
if err != nil {
|
||||
break // sql.ErrNoRows or real error — either way, done
|
||||
}
|
||||
claimed++
|
||||
go s.execute(ctx, task)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute()` changes:
|
||||
- Remove the `GetActiveRun` TOCTOU check — replaced by
|
||||
`CreateRunExclusive`.
|
||||
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
|
||||
But adoption also uses `CreateRunExclusive` semantics (transition
|
||||
only if still `queued`).
|
||||
- `advanceNextRun` still computes and sets `next_run_at`, making the
|
||||
task visible to future polls again.
|
||||
|
||||
### 1.4 — Remove `ListDue` from Interface
|
||||
|
||||
`ListDue` removed from `TaskStore` interface. Replaced by
|
||||
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
|
||||
`CreateRunExclusive` added alongside it.
|
||||
|
||||
Updated interface:
|
||||
```go
|
||||
type TaskStore interface {
|
||||
// ... existing CRUD ...
|
||||
|
||||
// Scheduler queries
|
||||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||||
SetNextRun(ctx context.Context, id string, nextRun interface{}) error
|
||||
SetLastRun(ctx context.Context, id string) error
|
||||
IncrementRunCount(ctx context.Context, id string) error
|
||||
|
||||
// Run history
|
||||
CreateRun(ctx context.Context, r *models.TaskRun) error
|
||||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
UpdateRun(ctx context.Context, id string, status string, ...) error
|
||||
TransitionRunStatus(ctx context.Context, id string, status string) error
|
||||
GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
|
||||
}
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
|
||||
- [ ] `CreateRunExclusive` — PG + SQLite
|
||||
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
|
||||
- [ ] Remove `ListDue` from interface + both stores
|
||||
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
|
||||
disjoint task sets)
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS2 — PG Ticket Store
|
||||
|
||||
Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
|
||||
implementation. The middleware already programs to the `TicketValidator`
|
||||
interface — the swap is clean.
|
||||
|
||||
### 2.1 — Store Interface
|
||||
|
||||
New interface in `store/`:
|
||||
|
||||
```go
|
||||
// TicketStore manages short-lived single-use WebSocket auth tickets.
|
||||
type TicketStore interface {
|
||||
Issue(ctx context.Context, userID string) (string, error)
|
||||
Validate(ctx context.Context, ticketID string) (string, bool)
|
||||
Reap(ctx context.Context) (int, error)
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 — PG Implementation
|
||||
|
||||
**Issue:**
|
||||
```sql
|
||||
INSERT INTO ws_tickets (id, user_id, expires_at)
|
||||
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
|
||||
```
|
||||
|
||||
Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).
|
||||
|
||||
**Validate (atomic consume):**
|
||||
```sql
|
||||
DELETE FROM ws_tickets
|
||||
WHERE id = $1 AND expires_at > NOW()
|
||||
RETURNING user_id
|
||||
```
|
||||
|
||||
Single statement — delete + return. If expired or already consumed,
|
||||
zero rows → `("", false)`.
|
||||
|
||||
**Reap (TTL cleanup):**
|
||||
```sql
|
||||
DELETE FROM ws_tickets WHERE expires_at <= NOW()
|
||||
```
|
||||
|
||||
Called by a system task or inline during poll. No dedicated goroutine
|
||||
needed — the scheduler's 30s tick can piggyback, or register as a
|
||||
lightweight system function.
|
||||
|
||||
### 2.3 — SQLite Implementation
|
||||
|
||||
Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
|
||||
instead of `NOW() + INTERVAL`).
|
||||
|
||||
### 2.4 — Adapter for Middleware Interface
|
||||
|
||||
The middleware `TicketValidator` interface is:
|
||||
```go
|
||||
type TicketValidator interface {
|
||||
Validate(ticketID string) (string, bool)
|
||||
}
|
||||
```
|
||||
|
||||
The store's `Validate` takes a `context.Context`. Thin adapter:
|
||||
|
||||
```go
|
||||
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
|
||||
type TicketValidatorAdapter struct {
|
||||
Store store.TicketStore
|
||||
}
|
||||
|
||||
func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return a.Store.Validate(ctx, ticketID)
|
||||
}
|
||||
```
|
||||
|
||||
### 2.5 — Wiring in `main.go`
|
||||
|
||||
Replace:
|
||||
```go
|
||||
ticketStore := events.NewTicketStore()
|
||||
defer ticketStore.Stop()
|
||||
```
|
||||
|
||||
With:
|
||||
```go
|
||||
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
|
||||
```
|
||||
|
||||
The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
|
||||
directly. The WS auth middleware receives `ticketAdapter` (satisfies
|
||||
`TicketValidator`). No reaper goroutine — use system function or
|
||||
scheduler piggyback.
|
||||
|
||||
### 2.6 — Reap Strategy
|
||||
|
||||
Option A: Register `ticket_reap` as a system function
|
||||
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
|
||||
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
|
||||
infrastructure. Cons: requires task to exist.
|
||||
|
||||
Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
|
||||
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
|
||||
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.
|
||||
|
||||
**Recommendation: Option B.** At 50 users, the table will have at most
|
||||
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
|
||||
free. Revisit if ticket volume grows.
|
||||
|
||||
### 2.7 — Delete `events/tickets.go`
|
||||
|
||||
The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
|
||||
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
|
||||
lives in `events/` (or alongside the middleware — whichever avoids
|
||||
import cycles).
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `store.TicketStore` interface
|
||||
- [ ] PG implementation + SQLite implementation
|
||||
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
|
||||
`TicketValidator` interface
|
||||
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
|
||||
- [ ] Inline reap in scheduler poll
|
||||
- [ ] Delete `events/tickets.go`
|
||||
- [ ] Add `Tickets` field to `store.Stores`
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS3 — PG Rate Limiter
|
||||
|
||||
Replace the in-memory token bucket (`middleware.RateLimiter`) with a
|
||||
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
|
||||
burst 8), so transaction volume is trivially low.
|
||||
|
||||
### 3.1 — Store Interface
|
||||
|
||||
```go
|
||||
// RateLimitStore manages distributed rate limit counters.
|
||||
type RateLimitStore interface {
|
||||
// Allow checks if a request is within the rate limit.
|
||||
// Returns (allowed bool, tokensRemaining float64).
|
||||
// Atomically increments the counter if allowed.
|
||||
Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)
|
||||
|
||||
// Cleanup removes expired windows.
|
||||
Cleanup(ctx context.Context, maxAge time.Duration) error
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 — PG Implementation: Sliding-Window Token Bucket
|
||||
|
||||
The in-memory implementation uses a classic token bucket (refill based
|
||||
on elapsed time). The PG version approximates this with time-bucketed
|
||||
counters.
|
||||
|
||||
**`Allow` — atomic check-and-increment:**
|
||||
|
||||
```sql
|
||||
-- Upsert the current window's counter and check burst limit.
|
||||
-- Window = current second (truncated).
|
||||
INSERT INTO rate_limit_counters (key, window, tokens)
|
||||
VALUES ($1, date_trunc('second', NOW()), 1)
|
||||
ON CONFLICT (key, window)
|
||||
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
|
||||
RETURNING tokens
|
||||
```
|
||||
|
||||
Go logic after the RETURNING:
|
||||
1. Query returns `tokens` (count in the current 1-second window).
|
||||
2. If `tokens > burst` → denied, return `(false, nil)`.
|
||||
3. Else → allowed.
|
||||
|
||||
This is a fixed-window approximation, not a sliding window. For auth
|
||||
rate limiting at 5 req/s burst 8, the practical difference is
|
||||
negligible — worst case allows 2× burst at window boundaries, which
|
||||
is acceptable for this use case. A true sliding window would require
|
||||
reading adjacent windows and interpolating, adding complexity for
|
||||
near-zero benefit at this scale.
|
||||
|
||||
**Cleanup:**
|
||||
```sql
|
||||
DELETE FROM rate_limit_counters
|
||||
WHERE window < NOW() - $1::interval
|
||||
```
|
||||
|
||||
Called alongside ticket reap in the scheduler poll loop.
|
||||
|
||||
### 3.3 — SQLite Implementation
|
||||
|
||||
Same logic, `datetime('now')` for window truncation. Single-process,
|
||||
so the PG atomicity guarantees are naturally satisfied.
|
||||
|
||||
### 3.4 — Middleware Swap
|
||||
|
||||
Replace `middleware.RateLimiter` struct with one backed by the store:
|
||||
|
||||
```go
|
||||
type RateLimiter struct {
|
||||
store store.RateLimitStore
|
||||
rate float64
|
||||
burst int
|
||||
}
|
||||
|
||||
func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
|
||||
return &RateLimiter{store: store, rate: rate, burst: burst}
|
||||
}
|
||||
|
||||
func (rl *RateLimiter) Limit() gin.HandlerFunc {
|
||||
return func(c *gin.Context) {
|
||||
key := "auth:" + c.ClientIP()
|
||||
allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
|
||||
if err != nil {
|
||||
// DB error — fail open (don't block auth on rate limit DB failure)
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
if !allowed {
|
||||
c.Header("Retry-After", "1")
|
||||
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
|
||||
"error": "rate limit exceeded",
|
||||
})
|
||||
return
|
||||
}
|
||||
c.Next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Fail-open policy:** if PG is down, allow the request. The auth
|
||||
endpoints have their own protections (bcrypt, lockout). Blocking
|
||||
legitimate logins because the rate limit table is unreachable is worse
|
||||
than allowing a brief burst.
|
||||
|
||||
### 3.5 — Wiring in `main.go`
|
||||
|
||||
Replace:
|
||||
```go
|
||||
authLimiter := middleware.NewRateLimiter(5, 8)
|
||||
```
|
||||
|
||||
With:
|
||||
```go
|
||||
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
|
||||
```
|
||||
|
||||
Delete the in-memory cleanup goroutine (replaced by scheduler reap).
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `store.RateLimitStore` interface
|
||||
- [ ] PG implementation + SQLite implementation
|
||||
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
|
||||
- [ ] `main.go` wiring
|
||||
- [ ] Add `RateLimits` field to `store.Stores`
|
||||
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
|
||||
- [ ] Delete in-memory visitor map + cleanup goroutine
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS4 — WebSocket Cross-Pod Delivery
|
||||
|
||||
The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
|
||||
already implemented and handles room-scoped events correctly. The gap:
|
||||
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
|
||||
to local connection send channels. Users on other pods are never reached.
|
||||
|
||||
### Affected call sites (7 total)
|
||||
|
||||
| File | Context |
|
||||
|------|---------|
|
||||
| `main.go` | DM typing indicators |
|
||||
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
|
||||
| `handlers/stream_loop.go` | SSE stream events |
|
||||
| `handlers/tool_loop.go` | Browser tool call delivery |
|
||||
| `notifications/service.go` | Push notification delivery |
|
||||
|
||||
### 4.1 — Add `TargetUserID` to Event
|
||||
|
||||
```go
|
||||
type Event struct {
|
||||
Label string `json:"event"`
|
||||
Room string `json:"room,omitempty"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
Ts int64 `json:"ts"`
|
||||
|
||||
// Server-side metadata (not serialized to clients)
|
||||
SenderID string `json:"-"`
|
||||
ConnID string `json:"-"`
|
||||
TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
|
||||
}
|
||||
```
|
||||
|
||||
`TargetUserID` is included in the JSON serialization for `pg_broadcast`
|
||||
(so the remote pod knows who to deliver to) but stripped before sending
|
||||
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
|
||||
path, or use a separate internal envelope.
|
||||
|
||||
**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
|
||||
harmless if clients see it (they already ignore unknown fields), and
|
||||
avoids a second serialization path. The field is empty for room-scoped
|
||||
events and only populated for targeted delivery.
|
||||
|
||||
### 4.2 — Update WS Subscriber Filter
|
||||
|
||||
In `subscribeToBus`, add target filtering:
|
||||
|
||||
```go
|
||||
// Targeted delivery: if event has a target user, only deliver to that user
|
||||
if e.TargetUserID != "" && e.TargetUserID != c.userID {
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
This goes before the room filter. Targeted events skip room filtering
|
||||
entirely (they're user-scoped, not room-scoped).
|
||||
|
||||
### 4.3 — Rewrite `SendToUser` → `PublishToUser`
|
||||
|
||||
New method on `Hub`:
|
||||
|
||||
```go
|
||||
// PublishToUser sends an event to a specific user via the bus.
|
||||
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
|
||||
func (h *Hub) PublishToUser(userID string, event Event) {
|
||||
event.TargetUserID = userID
|
||||
h.bus.Publish(event)
|
||||
}
|
||||
```
|
||||
|
||||
This replaces all 7 `hub.SendToUser` call sites. The event flows:
|
||||
|
||||
1. `hub.PublishToUser(userID, evt)`
|
||||
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
|
||||
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
|
||||
4. `broadcastHook` → `pg_notify` → remote pod
|
||||
5. Remote pod: `publishLocal(evt)` → `subscribeToBus` filter → deliver
|
||||
|
||||
### 4.4 — Keep `SendToUser` as Local-Only Optimization
|
||||
|
||||
Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
|
||||
`PublishToUser` can optionally try local delivery first (fast path)
|
||||
and only broadcast if the user isn't connected locally. But this
|
||||
optimization is premature at 50 users — just always go through the
|
||||
bus. Revisit post-MVP if latency matters.
|
||||
|
||||
### 4.5 — Audit: `tool.call` + `WaitFor`
|
||||
|
||||
`tool.call.*` events use `DirToClient` routing and are delivered via
|
||||
`SendToUser`. After the rewrite, they go through the bus and will
|
||||
broadcast to all pods. The `WaitFor` on the originating pod still
|
||||
works — `tool.result.*` comes back from the client on the same pod
|
||||
(the WS connection is sticky to a pod).
|
||||
|
||||
Verify: if the user's browser tool is connected to pod-2, but the
|
||||
completion handler calling `WaitFor` is on pod-1, the tool call event
|
||||
needs to reach pod-2 (via pg_broadcast), and the result needs to come
|
||||
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
|
||||
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
|
||||
the broadcast hook skips it. **This needs to change to `DirBoth`** so
|
||||
tool results cross pod boundaries for `WaitFor` to work.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `TargetUserID` field on `Event`
|
||||
- [ ] `subscribeToBus` target filter
|
||||
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
|
||||
- [ ] Update `tool.result.*` routing to `DirBoth`
|
||||
- [ ] Rename `SendToUser` → `sendToUserLocal` (unexported)
|
||||
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS5 — Health Probes + Helm Validation
|
||||
|
||||
### 5.1 — Readiness Probe Refinement
|
||||
|
||||
Current health endpoint: basic HTTP 200. Add a PG ping:
|
||||
|
||||
```go
|
||||
func (h *HealthHandler) Readiness(c *gin.Context) {
|
||||
ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
|
||||
defer cancel()
|
||||
if err := database.DB.PingContext(ctx); err != nil {
|
||||
c.JSON(503, gin.H{"error": "database unavailable"})
|
||||
return
|
||||
}
|
||||
c.JSON(200, gin.H{"status": "ok"})
|
||||
}
|
||||
```
|
||||
|
||||
Wire as `/healthz/ready` (separate from the existing `/healthz/live`
|
||||
liveness probe). Kubernetes pulls the pod from the service on readiness
|
||||
failure — new requests route to healthy replicas.
|
||||
|
||||
### 5.2 — Helm Changes
|
||||
|
||||
`chart/values.yaml`:
|
||||
```yaml
|
||||
backend:
|
||||
replicaCount: 2 # was 1
|
||||
```
|
||||
|
||||
`chart/templates/deployment-backend.yaml` additions:
|
||||
|
||||
**Pod anti-affinity** — spread replicas across nodes:
|
||||
```yaml
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: switchboard-backend
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
`preferredDuring` (not `requiredDuring`) — if fewer nodes than
|
||||
replicas, pods still schedule on the same node (degraded HA is better
|
||||
than no scheduling).
|
||||
|
||||
**Readiness probe:**
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz/ready
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
### 5.3 — Startup Jitter
|
||||
|
||||
Add random jitter (0–15s) to the scheduler's initial tick to stagger
|
||||
replica polling:
|
||||
|
||||
```go
|
||||
func (s *Scheduler) Run() {
|
||||
// ...
|
||||
jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
|
||||
time.Sleep(jitter)
|
||||
log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Not strictly necessary with `SKIP LOCKED` (contention is harmless),
|
||||
but reduces unnecessary lock acquisition at startup.
|
||||
|
||||
### 5.4 — Validation Checklist
|
||||
|
||||
Manual validation on the `gobha-ai-chat` cluster:
|
||||
|
||||
- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
|
||||
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
|
||||
- [ ] Create a cron task (every 1 min), confirm only one run per tick
|
||||
- [ ] WebSocket: connect on pod-1, send message that completes on
|
||||
pod-2, verify streaming events arrive
|
||||
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
|
||||
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
|
||||
burst regardless of which replica serves each request
|
||||
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
|
||||
continues serving + picks up scheduler duties immediately
|
||||
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `/healthz/ready` endpoint with PG ping
|
||||
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
|
||||
- [ ] Scheduler startup jitter
|
||||
- [ ] Validation checklist executed and documented
|
||||
- [ ] CI green, tag `v0.32.0`
|
||||
|
||||
---
|
||||
|
||||
## Summary: Store Interface Additions
|
||||
|
||||
```go
|
||||
type Stores struct {
|
||||
// ... existing fields ...
|
||||
Tickets TicketStore // v0.32.0: WS auth tickets (PG-backed)
|
||||
RateLimits RateLimitStore // v0.32.0: Distributed rate limiting
|
||||
}
|
||||
```
|
||||
|
||||
`TaskStore` changes:
|
||||
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
|
||||
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
|
||||
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`
|
||||
|
||||
---
|
||||
|
||||
## Changeset Sequence
|
||||
|
||||
| CS | Scope | Key Files | Gate |
|
||||
|----|-------|-----------|------|
|
||||
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
|
||||
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
|
||||
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
|
||||
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
|
||||
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
|
||||
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |
|
||||
|
||||
---
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
|
||||
(rate limiter — auth-only, fail-open).
|
||||
|
||||
**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
|
||||
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
|
||||
`--replicas=1`).
|
||||
|
||||
**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
|
||||
handlers, completion, notifications). The `tool.result` routing change
|
||||
(`DirFromClient` → `DirBoth`) needs careful testing — if tool results
|
||||
broadcast to all pods, the non-originating pod's `WaitFor` must not
|
||||
accidentally consume the result. Verify: `WaitFor` subscribes to
|
||||
`tool.result.{specific-id}` (exact label match), so only the waiting
|
||||
goroutine receives it. Safe.
|
||||
|
||||
**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
|
||||
always a safe rollback. Each CS should leave CI green at single-replica.
|
||||
238
docs/archive/DESIGN-0.33.0.md
Normal file
238
docs/archive/DESIGN-0.33.0.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# DESIGN-0.33.0 — Observability
|
||||
|
||||
Operate the platform without reading Go source code. Six changesets
|
||||
covering structured logging, Prometheus metrics, OpenAPI docs, Grafana
|
||||
dashboards, alerting rules, and a built-in admin dashboard.
|
||||
|
||||
Depends on: v0.32.0 (Multi-Replica HA, current HEAD).
|
||||
|
||||
**Design decision:** Zero new database migrations. All metrics are
|
||||
in-memory per-pod, scraped by Prometheus. Existing tables
|
||||
(`provider_health`, `tool_health_windows`, `usage_log`, `audit_log`)
|
||||
continue serving the admin dashboard via stores.
|
||||
|
||||
**Library choices:**
|
||||
- `log/slog` (Go 1.22 stdlib) — zero-dep structured logging
|
||||
- `prometheus/client_golang` v1.20+ — canonical Go Prometheus client
|
||||
- Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
|
||||
- Swagger UI v5 via CDN — single HTML embed, zero build step
|
||||
|
||||
---
|
||||
|
||||
## What Already Existed
|
||||
|
||||
These required no structural changes, only instrumentation:
|
||||
|
||||
- **Health accumulator** — in-memory provider+tool metrics, 60s DB flush
|
||||
- **`Hub.ConnCount()`** — active WebSocket connection count
|
||||
- **`database.DB.Stats()`** — Go stdlib DB pool metrics
|
||||
- **K8s probes** — `/healthz/ready` + `/healthz/live` in Helm chart
|
||||
- **Middleware skip paths** — `/metrics` already in `SkipPaths` list
|
||||
- **`google/uuid`** — already in go.mod for request ID generation
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Structured Logging + Request ID
|
||||
|
||||
Foundation changeset. All subsequent work emits structured logs.
|
||||
|
||||
### `server/logging/logger.go` (new)
|
||||
|
||||
`Init(format, level)` configures the global `slog` logger:
|
||||
- `format=json` → `slog.NewJSONHandler(os.Stdout, opts)`
|
||||
- `format=text` → `slog.NewTextHandler(os.Stdout, opts)`
|
||||
- Levels: debug, info, warn, error
|
||||
|
||||
### `server/middleware/request_id.go` (new)
|
||||
|
||||
Generates UUID per request. Reuses existing `google/uuid`.
|
||||
Honors inbound `X-Request-Id` header for trace propagation.
|
||||
Sets `request_id` in Gin context and echoes header on response.
|
||||
|
||||
### Middleware chain rewrite
|
||||
|
||||
Changed `gin.Default()` → `gin.New()` with explicit chain:
|
||||
|
||||
```
|
||||
RequestID → Prometheus → Logger → Recovery → CORS
|
||||
```
|
||||
|
||||
Logger middleware rewritten from `log.Printf` to `slog.Info("request", ...)`
|
||||
with structured fields: method, path, status, latency_ms, client_ip,
|
||||
request_id, user_id.
|
||||
|
||||
### Config additions
|
||||
|
||||
- `LogFormat` (env: `LOG_FORMAT`, default: `text`)
|
||||
- `LogLevel` (env: `LOG_LEVEL`, default: `info`)
|
||||
|
||||
---
|
||||
|
||||
## CS1 — Prometheus `/metrics` Endpoint
|
||||
|
||||
### `server/metrics/metrics.go` (new)
|
||||
|
||||
All metric definitions in one file using `promauto`:
|
||||
|
||||
| Metric | Type | Labels |
|
||||
|--------|------|--------|
|
||||
| `switchboard_http_requests_total` | Counter | method, path_pattern, status |
|
||||
| `switchboard_http_request_duration_seconds` | Histogram | method, path_pattern |
|
||||
| `switchboard_websocket_connections` | Gauge | — |
|
||||
| `switchboard_completion_tokens_total` | Counter | direction, model_id |
|
||||
| `switchboard_completions_total` | Counter | provider_config_id, model_id, status |
|
||||
| `switchboard_completion_duration_seconds` | Histogram | provider_config_id, model_id |
|
||||
| `switchboard_provider_status` | Gauge | provider_config_id |
|
||||
| `switchboard_db_open_connections` | Gauge | — |
|
||||
| `switchboard_db_in_use_connections` | Gauge | — |
|
||||
| `switchboard_db_idle_connections` | Gauge | — |
|
||||
| `switchboard_db_wait_count` | Gauge | — |
|
||||
| `switchboard_db_wait_duration_seconds` | Gauge | — |
|
||||
| `switchboard_task_executions_total` | Counter | status |
|
||||
| `switchboard_sandbox_executions_total` | Counter | entry_point, status |
|
||||
|
||||
### `server/metrics/db_collector.go` (new)
|
||||
|
||||
Background goroutine reads `database.DB.Stats()` every 15s, updates
|
||||
Prometheus gauges.
|
||||
|
||||
### `server/middleware/prometheus.go` (new)
|
||||
|
||||
Gin middleware. Uses `c.FullPath()` for `path_pattern` label to avoid
|
||||
cardinality explosion from path parameters (e.g. `/channels/:id`
|
||||
not `/channels/abc123`).
|
||||
|
||||
### Instrumentation points
|
||||
|
||||
- `events/ws.go` — `WebSocketConnections.Inc()/Dec()` on connect/disconnect
|
||||
- `handlers/completion.go` — token counters in `logUsage()`, completion
|
||||
duration/status in `recordHealth()`
|
||||
- `handlers/stream_loop.go` — same instrumentation in `recordHealthFn()`
|
||||
- `health/accumulator.go` — `ProviderStatus` gauge updated in `flush()`
|
||||
|
||||
### Route
|
||||
|
||||
`/metrics` mounted via `promhttp.Handler()` (no auth, standard scraping).
|
||||
|
||||
---
|
||||
|
||||
## CS2 — OpenAPI Spec + Swagger UI
|
||||
|
||||
### `server/static/openapi.yaml` (new)
|
||||
|
||||
Hand-curated OpenAPI 3.0.3 spec covering core API groups:
|
||||
Auth, Channels, Completions, Health, Metrics, WebSocket Tickets,
|
||||
Admin (stats, health, usage, dashboard).
|
||||
|
||||
### `server/static/swagger.html` (new)
|
||||
|
||||
Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at
|
||||
`openapi.yaml`. Zero build step.
|
||||
|
||||
### Routes
|
||||
|
||||
- `GET /api/docs` → serves embedded `swagger.html`
|
||||
- `GET /api/docs/openapi.yaml` → serves embedded `openapi.yaml`
|
||||
|
||||
Both files embedded via `//go:embed` directives.
|
||||
|
||||
---
|
||||
|
||||
## CS3 — Grafana Dashboard + Alerting Rules + Helm
|
||||
|
||||
Static files + Helm templates. No Go code changes.
|
||||
|
||||
### `chart/dashboards/switchboard-overview.json` (new)
|
||||
|
||||
Grafana dashboard with 10 panels:
|
||||
- Request rate, error rate, latency percentiles
|
||||
- WebSocket connections, completion rate/latency by provider
|
||||
- Tokens/min by model, provider status
|
||||
- DB connection pool, task executions
|
||||
|
||||
Template variables: `$datasource`, `$namespace`, `$pod`.
|
||||
|
||||
### `chart/alerting/switchboard-rules.yaml` (new)
|
||||
|
||||
Source PrometheusRule with 6 alerts:
|
||||
|
||||
| Alert | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| SwitchboardPodRestart | Restarts in 1h | warning |
|
||||
| SwitchboardProviderDown | Status > 2 for 5m | critical |
|
||||
| SwitchboardDBPoolExhaustion | In-use/open > 80% for 5m | warning |
|
||||
| SwitchboardHighErrorRate | 5xx rate > 5% for 5m | warning |
|
||||
| SwitchboardTaskFailureRate | Error rate > 25% for 10m | warning |
|
||||
| SwitchboardNoCompletions | Zero completions for 15m | critical |
|
||||
|
||||
### Helm templates (new, all gated `enabled: false` by default)
|
||||
|
||||
- `chart/templates/servicemonitor.yaml` — ServiceMonitor CRD
|
||||
- `chart/templates/grafana-dashboard-configmap.yaml` — ConfigMap for
|
||||
Grafana sidecar discovery
|
||||
- `chart/templates/prometheusrule.yaml` — PrometheusRule CRD
|
||||
|
||||
### Helm modifications
|
||||
|
||||
- `chart/values.yaml` — `logging` section + `monitoring` section
|
||||
- `chart/templates/configmap.yaml` — `LOG_FORMAT`, `LOG_LEVEL` env vars
|
||||
- `chart/templates/services.yaml` — Prometheus scrape annotations
|
||||
|
||||
---
|
||||
|
||||
## CS4 — Admin Observability Dashboard
|
||||
|
||||
Built-in admin page for real-time health without Grafana dependency.
|
||||
|
||||
### `server/handlers/dashboard_admin.go` (new)
|
||||
|
||||
`GET /api/v1/admin/dashboard` aggregating:
|
||||
- Provider health summaries (from `HealthStore`)
|
||||
- 24h usage totals (from `UsageStore`)
|
||||
- DB pool stats (`database.DB.Stats()`)
|
||||
- WebSocket connection count (`Hub.ConnCount()`)
|
||||
- Recent errors (from `AuditStore`)
|
||||
- Process uptime
|
||||
|
||||
### Frontend
|
||||
|
||||
- `SCAFFOLDING.dashboard` — stat cards grid + content container
|
||||
- `ADMIN_LOADERS.dashboard` → `UI.loadAdminDashboard()`
|
||||
- `ADMIN_SECTIONS.monitoring` — `dashboard` added as first section
|
||||
- Dashboard auto-refreshes every 30s
|
||||
- CSS: provider grid, DB pool bar gauge, error list
|
||||
|
||||
### Admin template
|
||||
|
||||
Monitoring tab now lands on `/admin/dashboard` instead of `/admin/usage`.
|
||||
|
||||
---
|
||||
|
||||
## CS5 — Wiring, docker-compose, ICD Tests
|
||||
|
||||
### docker-compose.yml
|
||||
|
||||
Added `LOG_FORMAT` and `LOG_LEVEL` env vars with defaults.
|
||||
|
||||
### ICD tests (`crud/observability.js`)
|
||||
|
||||
| Test | Assertion |
|
||||
|------|-----------|
|
||||
| `GET /metrics` | 200, contains `switchboard_*` metrics |
|
||||
| `GET /api/docs` | 200, contains `swagger-ui` |
|
||||
| `GET /api/docs/openapi.yaml` | 200, contains `openapi:` and `paths:` |
|
||||
| X-Request-Id generation | Response includes 36-char UUID header |
|
||||
| X-Request-Id passthrough | Client-sent header echoed back |
|
||||
| `GET /admin/dashboard` | 200, has `uptime`, `ws_connections`, `provider_health` |
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. `LOG_FORMAT=json docker compose up` → JSON log lines on stdout
|
||||
2. `curl localhost:8080/metrics | grep switchboard_` → all metrics present
|
||||
3. `curl -I localhost:8080/api/v1/channels` → `X-Request-Id` header
|
||||
4. `/api/docs` in browser → Swagger UI renders
|
||||
5. `/admin` → Monitoring → Dashboard shows live stat cards
|
||||
6. `helm lint chart/ --set monitoring.serviceMonitor.enabled=true` → passes
|
||||
7. ICD test suite passes (existing + observability tests)
|
||||
217
docs/archive/DESIGN-0.34.0.md
Normal file
217
docs/archive/DESIGN-0.34.0.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# DESIGN-0.34.0 — Data Portability
|
||||
|
||||
Export your data, import it elsewhere, delete your account. Six
|
||||
changesets covering user export, user import, GDPR delete, admin
|
||||
team export/import, ChatGPT import, and K8s scheduled backups.
|
||||
|
||||
Depends on: v0.33.0 (Observability, current HEAD).
|
||||
|
||||
**Design decision:** One new migration (021) — indexes only, no new
|
||||
tables. The export archive is a ZIP file (`.switchboard` extension)
|
||||
containing a JSON manifest and per-entity JSON files. Sensitive fields
|
||||
(password hashes, encrypted keys, storage keys) are stripped at export
|
||||
time via entity sanitization.
|
||||
|
||||
**Archive format:** `.switchboard` ZIP with `format_version=1`:
|
||||
```
|
||||
manifest.json
|
||||
data/users.json
|
||||
data/channels.json
|
||||
data/messages.json
|
||||
data/channel_participants.json
|
||||
data/channel_models.json
|
||||
data/channel_cursors.json
|
||||
data/notes.json
|
||||
data/note_links.json
|
||||
data/memories.json
|
||||
data/projects.json
|
||||
data/project_channels.json
|
||||
data/project_knowledge_bases.json
|
||||
data/project_notes.json
|
||||
data/workspaces.json
|
||||
data/workspace_files.json
|
||||
data/folders.json
|
||||
data/user_settings.json
|
||||
data/notification_preferences.json
|
||||
data/usage_entries.json
|
||||
data/persona_groups.json
|
||||
data/persona_group_members.json
|
||||
data/files.json
|
||||
files/{file_id}/{filename}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Already Existed
|
||||
|
||||
- **Package export** — `.pkg` ZIP format, `package_export.go`
|
||||
- **Markdown export** — pandoc-based, `export.go`
|
||||
- **Workspace archive** — ZIP/tar.gz with size limits, `workspace/archive.go`
|
||||
- **Soft deletes** — `deleted_at` on messages
|
||||
- **Audit trail** — `audit_logs` table
|
||||
- **39 store interfaces** covering all entity types
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Export Format + User Export
|
||||
|
||||
Foundation changeset. Defines the archive format and ExportStore interface.
|
||||
|
||||
### `server/export/format.go` (new)
|
||||
|
||||
Archive types: `Manifest`, `ManifestScope`, `ArchiveWriter`, `ArchiveReader`.
|
||||
Size limits: 500 MB archive, 100 MB per file, 10K files max.
|
||||
Key methods: `WriteManifest`, `WriteEntityJSON`, `WriteFile`,
|
||||
`ReadManifest`, `ReadEntityJSON`, `FileEntries`.
|
||||
|
||||
### `server/export/entities.go` (new)
|
||||
|
||||
Per-entity sanitization functions. Export types strip sensitive fields:
|
||||
- `ExportChannel` — removes `provider_config_id`
|
||||
- `ExportMessage` — no changes needed
|
||||
- `ExportWorkspace` — removes `root_path`, git credentials
|
||||
- `ExportUsageEntry` — removes `provider_config_id`
|
||||
- Users — removes `password_hash`, `encrypted_uek`, `uek_salt`
|
||||
|
||||
### `server/store/interfaces.go` (modified)
|
||||
|
||||
Added `Export ExportStore` to `Stores` struct. The `ExportStore`
|
||||
interface provides 21 user-scoped read methods, 20 import methods
|
||||
(returning `(imported, skipped, error)` counts), 14 team-scoped
|
||||
read methods, and 4 GDPR methods.
|
||||
|
||||
### `server/store/{postgres,sqlite}/export.go` (new)
|
||||
|
||||
Full implementations for both dialects (~1400-1500 lines each).
|
||||
Import uses `INSERT ON CONFLICT DO NOTHING` (PG) / `INSERT OR IGNORE`
|
||||
(SQLite) for UUID-based dedup.
|
||||
|
||||
### `server/handlers/export_data.go` (new)
|
||||
|
||||
`DataExportHandler.ExportMyData` — `GET /api/v1/export/me`.
|
||||
Streams ZIP directly to response using `zip.NewWriter(c.Writer)`.
|
||||
Includes file blobs from object store.
|
||||
|
||||
### Migration 021
|
||||
|
||||
Indexes only — `idx_messages_channel_active`, `idx_channels_user_id`,
|
||||
`idx_files_user_id`. PG version uses partial index
|
||||
(`WHERE deleted_at IS NULL`).
|
||||
|
||||
---
|
||||
|
||||
## CS1 — User Import
|
||||
|
||||
### `server/handlers/import_data.go` (new)
|
||||
|
||||
`DataImportHandler.ImportMyData` — `POST /api/v1/import/me`.
|
||||
|
||||
Opens archive, validates manifest `format_version`, reads entities
|
||||
in FK-dependency order, remaps `user_id` for cross-instance import.
|
||||
Returns `{imported: {channels: N, ...}, skipped: {...}, errors: [...]}`.
|
||||
|
||||
Import order: Folders → Projects → Workspaces → Channels →
|
||||
Participants/Models/Cursors → Messages → Notes → Note Links →
|
||||
Memories → Project links → Workspace files → Files + blobs →
|
||||
Settings → Persona groups.
|
||||
|
||||
---
|
||||
|
||||
## CS2 — GDPR Account Delete
|
||||
|
||||
### `server/handlers/gdpr.go` (new)
|
||||
|
||||
`GDPRHandler.DeleteMyAccount` — `DELETE /api/v1/me`.
|
||||
|
||||
Requires `{"confirm": "DELETE", "password": "..."}`. Verifies
|
||||
password, prevents last-admin deletion, then:
|
||||
|
||||
1. Soft-deletes messages, archives channels
|
||||
2. Hard-deletes: notes, memories, workspaces, files, projects, settings
|
||||
3. Revokes tokens (refresh + WS tickets)
|
||||
4. Anonymizes user: `deleted-user-{sha256(id)[:12]}`
|
||||
5. Writes audit log entry
|
||||
|
||||
---
|
||||
|
||||
## CS3 — Admin Team Export/Import
|
||||
|
||||
`DataExportHandler.ExportTeam` — `GET /api/v1/admin/teams/:id/export`
|
||||
`DataImportHandler.ImportTeam` — `POST /api/v1/admin/teams/:id/import`
|
||||
|
||||
Same archive format, scoped to team entities (channels, personas,
|
||||
knowledge bases, workflows, groups, projects, resource grants).
|
||||
Team import skips members whose `user_id` doesn't exist on target.
|
||||
|
||||
---
|
||||
|
||||
## CS4 — ChatGPT Import
|
||||
|
||||
### `server/export/chatgpt.go` (new)
|
||||
|
||||
Parses ChatGPT `conversations.json` format. Each conversation maps
|
||||
to one channel (type=direct). The `mapping` DAG is walked depth-first
|
||||
to produce ordered messages with `parent_id` and `sibling_index`.
|
||||
|
||||
Role mapping: `user`→user, `assistant`→assistant, `system`→system,
|
||||
`tool`→tool. Content: `parts[]` joined with `\n`, non-string parts
|
||||
skipped. Timestamps: `float64` epoch → `time.Time`.
|
||||
|
||||
`DataImportHandler.ImportChatGPT` — `POST /api/v1/import/chatgpt`.
|
||||
Accepts multipart upload of `conversations.json` or ZIP containing it.
|
||||
|
||||
---
|
||||
|
||||
## CS5 — K8s Backup + ICD Tests
|
||||
|
||||
### `chart/templates/cronjob-backup.yaml` (new)
|
||||
|
||||
CronJob running `pg_dump | gzip` on schedule (default daily 02:00 UTC).
|
||||
Optional S3 upload via AWS CLI. Local retention pruning. Gated on
|
||||
`backup.enabled=true` and `database.driver=postgres`.
|
||||
|
||||
### `chart/templates/pvc-backup.yaml` (new)
|
||||
|
||||
Conditional PVC for local backup storage (when `backup.persistence.enabled`).
|
||||
|
||||
### `chart/values.yaml` (modified)
|
||||
|
||||
New `backup:` section with schedule, retention, S3 config, persistence,
|
||||
and resource limits.
|
||||
|
||||
### ICD Tests
|
||||
|
||||
~30 new tests in `crud/portability.js` covering:
|
||||
- Export download + archive validation (ZIP structure, manifest, entities)
|
||||
- Sensitive field exclusion verification
|
||||
- Import re-import (dedup validation)
|
||||
- Invalid archive rejection
|
||||
- ChatGPT import (valid, invalid JSON, empty array)
|
||||
- ChatGPT channel/message verification
|
||||
- GDPR delete flow (register → create data → delete → verify lockout)
|
||||
- GDPR validation (missing confirm, wrong password)
|
||||
- Admin team export/import
|
||||
|
||||
---
|
||||
|
||||
## New Routes
|
||||
|
||||
| Method | Path | Auth | Handler |
|
||||
|--------|------|------|---------|
|
||||
| GET | `/api/v1/export/me` | JWT | `DataExportHandler.ExportMyData` |
|
||||
| POST | `/api/v1/import/me` | JWT | `DataImportHandler.ImportMyData` |
|
||||
| POST | `/api/v1/import/chatgpt` | JWT | `DataImportHandler.ImportChatGPT` |
|
||||
| DELETE | `/api/v1/me` | JWT | `GDPRHandler.DeleteMyAccount` |
|
||||
| GET | `/api/v1/admin/teams/:id/export` | Admin | `DataExportHandler.ExportTeam` |
|
||||
| POST | `/api/v1/admin/teams/:id/import` | Admin | `DataImportHandler.ImportTeam` |
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. `cd server && go build ./...` — all changesets compile clean
|
||||
2. `helm template ./chart --set backup.enabled=true` — CronJob renders
|
||||
3. ICD test suite: ~615 tests (585 existing + ~30 portability)
|
||||
4. Manual smoke: export → inspect ZIP → re-import → verify dedup
|
||||
5. GDPR: delete → verify login fails → inspect anonymized DB record
|
||||
6. Sensitive fields: no `password_hash`, `encrypted_uek`, `storage_key` in export
|
||||
148
docs/archive/DESIGN-0.35.0.md
Normal file
148
docs/archive/DESIGN-0.35.0.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# DESIGN-0.35.0 — Workflow Product
|
||||
|
||||
Transforms the workflow engine from a linear stage-runner into a
|
||||
product-grade automation tool. Seven changesets covering conditional
|
||||
routing, AI-triggered routing, data pipeline hooks, progressive forms,
|
||||
conditional fields, structured review, and a monitoring dashboard.
|
||||
|
||||
Depends on: v0.34.0 (Data Portability), v0.31.2 (Team Workflows),
|
||||
v0.29.0 (Starlark Sandbox).
|
||||
|
||||
## Schema Changes
|
||||
|
||||
Migrations modified in-place (DB rebuild required):
|
||||
|
||||
**005_channels.sql** — add `stage_entered_at` (PG: `TIMESTAMPTZ`,
|
||||
SQLite: `TEXT`). Tracks when the current stage was entered for SLA
|
||||
computation. Also add index `idx_channels_workflow_active` on
|
||||
`(workflow_id, workflow_status)` for monitoring queries.
|
||||
|
||||
**018_workflows.sql** — add `sla_seconds INTEGER` to `workflow_stages`.
|
||||
Add `review_comments` (PG: `JSONB DEFAULT '[]'`, SQLite: `TEXT
|
||||
DEFAULT '[]'`) to `workflow_assignments`.
|
||||
|
||||
No new migration files. No new tables.
|
||||
|
||||
## Conditional Routing Engine
|
||||
|
||||
**File:** `server/workflow/routing.go`
|
||||
|
||||
The existing advance logic hardcoded `nextStage = currentStage + 1`.
|
||||
The routing engine replaces this with condition evaluation:
|
||||
|
||||
1. Parse `transition_rules.conditions[]` from the departing stage
|
||||
2. Evaluate each condition against accumulated `stage_data`
|
||||
3. First match → resolve `target_stage` (by name or ordinal)
|
||||
4. No match → fallback to `ordinal + 1`
|
||||
|
||||
**Expression format:** `{field, op, value, target_stage}`. Operators:
|
||||
`eq`, `neq`, `gt`, `lt`, `gte`, `lte`, `in`, `contains`, `exists`,
|
||||
`not_exists`. Loose equality via `fmt.Sprintf("%v")` comparison.
|
||||
Numeric comparison extracts `float64` from both sides.
|
||||
|
||||
**Wiring:** `ResolveNextStage()` called in all 4 advance sites:
|
||||
handler (`workflow_instances.go`), tool (`tools/workflow.go`), forms
|
||||
auto-advance (`workflow_forms.go`), Starlark module
|
||||
(`sandbox/workflow_module.go`).
|
||||
|
||||
**Backward compatible:** no conditions = `ordinal + 1`.
|
||||
|
||||
## AI-Triggered Routing
|
||||
|
||||
**Tool:** `workflow_route` — persona calls with `{target_stage, reason}`.
|
||||
Uses `ResolveStageByName()` (case-insensitive). Records decision in
|
||||
`stage_data._route_history_latest`. Allows forward and backward jumps
|
||||
(loop stages for iterative correction).
|
||||
|
||||
**Starlark:** `workflow.route(channel_id, target_stage, reason)` for
|
||||
extension-driven escalation patterns. Requires `workflow.access`.
|
||||
|
||||
## on_advance Hook
|
||||
|
||||
**File:** `server/handlers/workflow_hooks.go`
|
||||
|
||||
Fires synchronously after `AdvanceWorkflowStage` succeeds. Configured
|
||||
per-stage in `transition_rules.on_advance: {package_id, entry_point}`.
|
||||
|
||||
Hook receives: `{stage_data, previous_stage, current_stage, channel_id}`.
|
||||
Returns: `{stage_data: {...}}` (enriched), `None` (no change), or
|
||||
`{error: "msg"}` (logged, not rolled back — future: rollback support).
|
||||
|
||||
The existing `http` Starlark module (v0.29.1) is available in hooks,
|
||||
enabling external API enrichment without new infrastructure.
|
||||
|
||||
## Progressive Forms
|
||||
|
||||
**Model:** `TypedFormTemplate.Fieldsets []FormFieldset` — optional
|
||||
array of `{label, fields[]}`. When present, top-level `fields` is
|
||||
replaced by flattened fieldset fields in `ParseTypedFormTemplate()`.
|
||||
|
||||
**Frontend:** `workflow.html` renders one fieldset at a time with
|
||||
step indicator, next/back navigation. All fields submitted together
|
||||
on the final step.
|
||||
|
||||
## Conditional Fields
|
||||
|
||||
**Model:** `FormField.Condition *FieldCondition` — `{when, op, value}`.
|
||||
Operators: `eq` (default), `neq`, `in`, `exists`.
|
||||
|
||||
**Server-side:** `ValidateFormData()` calls `evaluateFieldCondition()`
|
||||
before each field. Hidden fields are skipped (no required check, no
|
||||
type validation).
|
||||
|
||||
**Client-side:** `workflow.html` attaches `change`/`input` listeners
|
||||
to source fields. Target fields toggle `cond-hidden` CSS class.
|
||||
|
||||
## Structured Review
|
||||
|
||||
**Review comments:** `review_comments JSONB` on `workflow_assignments`.
|
||||
`POST /workflow-assignments/:id/comment` appends `{text, user_id,
|
||||
created_at}`. `GET /workflow-assignments/:id` returns full assignment.
|
||||
|
||||
**Review surface:** Side-by-side layout in `workflow.html` — left panel
|
||||
shows structured `stage_data` table (internal `_`-prefixed keys hidden),
|
||||
right panel has comment textarea + approve/reject buttons. Keyboard
|
||||
shortcuts: `Ctrl+Enter` approve, `Ctrl+Shift+Enter` reject.
|
||||
|
||||
## Monitoring Dashboard
|
||||
|
||||
**File:** `server/handlers/workflow_monitor.go`
|
||||
|
||||
**Endpoints:**
|
||||
- `GET /admin/workflows/monitor/instances` — all active instances
|
||||
- `GET /admin/workflows/monitor/funnel/:id` — stage counts
|
||||
- `GET /admin/workflows/monitor/stale?threshold_hours=N`
|
||||
- `GET /teams/:teamId/workflows/monitor/instances` — team-scoped
|
||||
|
||||
**SLA computation:** `stage_entered_at + sla_seconds` vs `NOW()`.
|
||||
Returned as `sla_remaining_seconds` (negative = breached) and
|
||||
`sla_breached` boolean. Computed at query time, not stored.
|
||||
|
||||
**Store:** New `ListByType(ctx, "workflow")` method on `ChannelStore`
|
||||
(both PG and SQLite). Monitoring queries iterate active channels,
|
||||
joining workflow definitions and stage metadata.
|
||||
|
||||
**Frontend:** `src/js/workflow-monitor.js` — auto-refresh every 30s,
|
||||
SLA color indicators (green > 50%, yellow > 20%, red), stale instance
|
||||
alert section.
|
||||
|
||||
## Workflow Branding
|
||||
|
||||
`WorkflowPageData.BrandingJSON` passes the workflow's `branding` JSON
|
||||
to the visitor template. Applied as:
|
||||
- `--accent` CSS custom property (from `accent_color`)
|
||||
- Logo image in header (from `logo_url`)
|
||||
- Tagline text below header (from `tagline`)
|
||||
|
||||
Previously branding was only on the landing page; now also on the
|
||||
active workflow surface.
|
||||
|
||||
## New Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server/workflow/routing.go` | Conditional routing engine |
|
||||
| `server/handlers/workflow_hooks.go` | on_advance hook dispatch |
|
||||
| `server/handlers/workflow_monitor.go` | Monitoring dashboard |
|
||||
| `src/js/workflow-monitor.js` | Admin monitoring frontend |
|
||||
| `packages/icd-test-runner/js/crud/workflow-product.js` | E2E tests |
|
||||
773
docs/archive/DESIGN-SURFACES.md
Normal file
773
docs/archive/DESIGN-SURFACES.md
Normal file
@@ -0,0 +1,773 @@
|
||||
# DESIGN — Surface & Extension Architecture
|
||||
|
||||
**Status:** Accepted (Phases 1–2 implemented in v0.25.1)
|
||||
**Scope:** Primitives, Components, Surfaces, Extension hooks, Themes
|
||||
**Depends on:** v0.22.5 (Go template engine), v0.22.7 (ChatPane), v0.25.1 (audit)
|
||||
**Informs:** v0.22.8+, EXTENSIONS.md rewrite, ARCHITECTURE.md update, ICD-API, ICD-SURFACE
|
||||
**Note:** For current package/surface authoring, see [PACKAGES.md](PACKAGES.md) and [EXTENSION-SURFACES.md](EXTENSION-SURFACES.md). For workflow surfaces, see [WORKFLOW-PACKAGES.md](WORKFLOW-PACKAGES.md).
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The UI is a four-layer architecture: Primitives, Components, Surfaces,
|
||||
and Banners. Each layer has a single responsibility and a strict
|
||||
dependency direction — primitives know nothing, components compose
|
||||
primitives, surfaces compose components, banners are platform chrome
|
||||
outside the surface boundary.
|
||||
|
||||
Extensions participate at every layer through six defined hooks.
|
||||
Admin-installed extensions are privileged (full DOM, direct API access,
|
||||
own routes). Themes are pure CSS custom property overrides that
|
||||
propagate through every layer uniformly.
|
||||
|
||||
This document defines the layer contracts, extension hook system,
|
||||
display content model, trust boundaries, and theme architecture.
|
||||
|
||||
---
|
||||
|
||||
## Definitions
|
||||
|
||||
A **surface** is a full-page layout — a composition of components with
|
||||
a route, a data loader, and a boot script. Chat is a surface. The
|
||||
editor is a surface. A custom triage intake form is a surface.
|
||||
|
||||
An **extension** is a package — a manifest plus code plus assets that
|
||||
participates in the platform through hooks. An extension can do one,
|
||||
some, or all of:
|
||||
|
||||
- Register tools the LLM can call (Hook 5: Tool Bridge)
|
||||
- Transform content during streaming (Hook 2: Stream Processing)
|
||||
- Enhance rendered messages (Hook 3: Post-Render)
|
||||
- Attach visual content to messages (Hook 4: Display Content)
|
||||
- Inject panels or sections into existing surfaces (Hook 6: Surface Injection)
|
||||
- Create entirely new surfaces with their own routes (Surface Registration)
|
||||
|
||||
The five **core surfaces** (Chat, Editor, Notes, Admin, Settings) exist
|
||||
without extensions. They are built from the same primitive and component
|
||||
layers that extensions use, but they are registered directly in Go
|
||||
rather than through a manifest.
|
||||
|
||||
An extension that creates a surface gets its own route (`/s/:slug`),
|
||||
its own data loader, and the full primitive/component library. An
|
||||
extension that only registers a post-render hook (like a block
|
||||
renderer) doesn't create any surfaces at all.
|
||||
|
||||
**Examples:**
|
||||
|
||||
| Extension | Creates Surface? | Hooks Used |
|
||||
|-----------|-----------------|------------|
|
||||
| Mermaid renderer | No | Post-Render |
|
||||
| KaTeX renderer | No | Post-Render |
|
||||
| Calculator tool | No | Tool Bridge |
|
||||
| Image Generator | Yes (`/s/gallery`) | Tool Bridge, Stream Processing, Post-Render, Display Content, Surface |
|
||||
| Custom Dashboard | Yes (`/s/dashboard`) | Surface, Surface Injection (admin section) |
|
||||
|
||||
---
|
||||
|
||||
## Existing Extension Mapping
|
||||
|
||||
The current extension mechanisms map directly into this architecture:
|
||||
|
||||
| Current Mechanism | Architecture Equivalent |
|
||||
|-------------------|------------------------|
|
||||
| Block renderers (`ctx.renderers.register()`) | Hook 3: Post-Render |
|
||||
| Tool bridge (`ctx.tools.register()`) | Hook 5: Tool Bridge |
|
||||
| `ctx.ui.toast()`, `ctx.ui.openPreview()` | Primitive layer (unchanged API) |
|
||||
| `ctx.ui.isDark()`, `ctx.ui.isMobile()` | Theme layer queries |
|
||||
| `ctx.surfaces.getCurrent()` | Returns `window.__SURFACE__` |
|
||||
|
||||
The manifest schema gains optional new fields (`hooks`, `surfaces`,
|
||||
`surface_injections`, `theme`). Existing fields retain their meaning.
|
||||
|
||||
`Extensions.boot()` is the single entry point for extension
|
||||
initialization on every surface. It loads manifests, registers
|
||||
renderers, and initializes tool bridges — same sequence, available
|
||||
on every surface rather than only chat.
|
||||
|
||||
---
|
||||
|
||||
## Layer Model
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Banner (top) │ ← Platform config
|
||||
├──────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Surface │ ← Layout + lifecycle
|
||||
│ ┌─────────────┬──────────────────────────┐ │
|
||||
│ │ Component │ Component │ │ ← Domain-aware
|
||||
│ │ (ChatPane) │ (NoteEditor) │ │
|
||||
│ │ ┌─────────┐ │ ┌────────┐ ┌──────────┐ │ │
|
||||
│ │ │Primitive│ │ │Primitv.│ │Primitive │ │ │ ← Atomic UI
|
||||
│ │ │ (input) │ │ │ (menu) │ │ (toggle) │ │ │
|
||||
│ │ └─────────┘ │ └────────┘ └──────────┘ │ │
|
||||
│ └─────────────┴──────────────────────────┘ │
|
||||
│ │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Banner (bottom) │ ← Platform config
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Four layers, strict dependency direction: Primitives know nothing.
|
||||
Components use Primitives. Surfaces compose Components. Banners are
|
||||
global chrome outside the surface boundary.
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: Primitives
|
||||
|
||||
Atomic UI elements. No domain logic. A toggle doesn't know if it's
|
||||
toggling a KB or a theme — it takes a label, a state, and a callback.
|
||||
|
||||
Styled entirely via CSS custom properties (the theme contract). Every
|
||||
primitive reads from the same property namespace, so theme changes
|
||||
propagate instantly.
|
||||
|
||||
### Catalog
|
||||
|
||||
| Primitive | Description |
|
||||
|-----------|-------------|
|
||||
| `Input` | Text, textarea, number, password, with label + validation |
|
||||
| `Select` | Dropdown, single or multi |
|
||||
| `Toggle` | Boolean switch with label |
|
||||
| `Checkbox` | Checkbox with label |
|
||||
| `Button` | Text button, variants: primary, secondary, danger, ghost |
|
||||
| `IconButton` | Icon-only button with tooltip |
|
||||
| `ColorPicker` | Color input with hex text field |
|
||||
| `Menu` | Dropdown or context menu, item list with icons + shortcuts |
|
||||
| `Dialog` | Modal: confirm, prompt, or custom form content |
|
||||
| `Toast` | Transient notification: success, error, warning, info |
|
||||
| `Badge` | Inline label: accent, success, danger, warning, muted |
|
||||
| `Avatar` | Image circle with upload affordance |
|
||||
| `Tabs` | Tab bar with content switching |
|
||||
| `FormGroup` | Label + input + validation message + help text |
|
||||
| `SectionHeader` | Titled section divider |
|
||||
| `EmptyState` | Placeholder with icon + message + action |
|
||||
| `Spinner` | Loading indicator |
|
||||
| `Table` | Sortable, paginated data table |
|
||||
|
||||
### Implementation
|
||||
|
||||
Each primitive is a factory function that returns a DOM element (or
|
||||
attaches to an existing one). No classes, no inheritance — just
|
||||
functions.
|
||||
|
||||
```js
|
||||
// Example — not prescriptive API, just the pattern:
|
||||
Primitives.toggle({ label: 'Auto-search', value: true, onChange: fn })
|
||||
Primitives.menu({ anchor: el, items: [...], onSelect: fn })
|
||||
Primitives.dialog({ title: 'Confirm', body: el, onConfirm: fn })
|
||||
```
|
||||
|
||||
Current locations: `ui-primitives.js` (factory functions, `esc()`,
|
||||
`createComponentRegistry()`, `componentMixin()`, `showConfirm()`,
|
||||
`showPrompt()`, `Theme`, `mountAvatarUpload()`)
|
||||
and Go template components (`model-select.html`, `team-select.html`,
|
||||
`file-upload.html`). These converge into one coherent set.
|
||||
|
||||
### Go Template Primitives
|
||||
|
||||
Some primitives need server-rendered initial state (e.g., model-select
|
||||
needs the model list, team-select needs the team list). These are Go
|
||||
template partials that render the HTML + initial data, then JS hydrates
|
||||
interactivity on load.
|
||||
|
||||
```html
|
||||
{{template "model-select" dict "ID" "channelModel" "Models" .Models "Type" "chat"}}
|
||||
```
|
||||
|
||||
The JS primitive attaches to the server-rendered DOM by ID, not by
|
||||
replacing it.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: Components
|
||||
|
||||
Domain-aware compositions of primitives. Each component owns its DOM
|
||||
subtree, manages its own state, and exposes a clean API for the surface
|
||||
to interact with.
|
||||
|
||||
### Catalog
|
||||
|
||||
| Component | Primitives Used | Domain |
|
||||
|-----------|----------------|--------|
|
||||
| `ChatPane` | Input, Button, Menu, Spinner, Toast | Channels, Messages, Streaming |
|
||||
| `NoteEditor` | Input (CM6), Menu, Tabs, Badge | Notes, Wikilinks, Folders |
|
||||
| `FileTree` | Menu (context), Button, Spinner | Workspaces, Files |
|
||||
| `ModelSelector` | Select, Badge, Spinner | Models, Capabilities, Health |
|
||||
| `PersonaPicker` | Select, Badge, Avatar | Personas, Scopes |
|
||||
| `KBPicker` | Toggle, Badge, Select | Knowledge Bases, Discoverability |
|
||||
| `ProjectSidebar` | Menu, Tabs, Badge, EmptyState | Projects, Channels, DnD |
|
||||
| `AuditLog` | Table, Select, Badge | Audit, Filtering, Pagination |
|
||||
| `CodeEditor` | Input (CM6), Select, Tabs | Workspaces, Languages |
|
||||
| `SettingsForm` | FormGroup, Toggle, Select, Button | User/Admin Settings |
|
||||
|
||||
### Instance Pattern
|
||||
|
||||
Components use `createComponentRegistry()` and `componentMixin()` from
|
||||
`ui-primitives.js` for shared lifecycle (instance tracking, event
|
||||
listener cleanup, destroy):
|
||||
|
||||
```js
|
||||
const ChatPane = {
|
||||
...createComponentRegistry('ChatPane'),
|
||||
|
||||
create(opts) {
|
||||
const instance = componentMixin({
|
||||
id: opts.id || 'pane-' + Date.now(),
|
||||
messagesEl: opts.messagesEl,
|
||||
inputEl: opts.inputEl,
|
||||
channelId: opts.channelId || null,
|
||||
// ... domain-specific state
|
||||
|
||||
// Component-specific teardown (called by mixin's destroy())
|
||||
_cleanup() {
|
||||
if (this._cmView) { this._cmView.destroy(); this._cmView = null; }
|
||||
},
|
||||
}, ChatPane);
|
||||
|
||||
ChatPane._register(instance.id, instance);
|
||||
return instance;
|
||||
},
|
||||
|
||||
// Custom queries beyond .get(id)
|
||||
forChannel(channelId) { /* ... */ },
|
||||
};
|
||||
```
|
||||
|
||||
The mixin provides `_on(el, event, handler)` for tracked event binding
|
||||
and `destroy()` which calls `_cleanup()` then removes all listeners and
|
||||
unregisters from the registry. Components that need extra teardown define
|
||||
`_cleanup()`.
|
||||
|
||||
Components can be instantiated multiple times on the same page (editor
|
||||
has a ChatPane, notes has a ChatPane — independent instances).
|
||||
|
||||
### Server-Rendered Shell + JS Hydration
|
||||
|
||||
Components have a Go template partial for the server-rendered scaffold:
|
||||
|
||||
```html
|
||||
{{template "chat-pane" dict "ID" "editor"}}
|
||||
```
|
||||
|
||||
This renders the DOM structure with predictable IDs. The JS component
|
||||
attaches to these IDs on DOMContentLoaded. No client-side DOM
|
||||
construction for the initial layout.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Surfaces
|
||||
|
||||
A surface is a full-page layout that composes components. Each surface
|
||||
declares:
|
||||
|
||||
1. **What components it uses** (ChatPane, NoteEditor, FileTree, etc.)
|
||||
2. **How they're arranged** (CSS grid/flex layout)
|
||||
3. **What data it needs on load** (Go data loader)
|
||||
4. **What JS runs on boot** (script block or module)
|
||||
|
||||
### Current Surfaces
|
||||
|
||||
| Surface | Components | Data Loader |
|
||||
|---------|-----------|-------------|
|
||||
| Chat | ChatPane, ProjectSidebar, ModelSelector, PersonaPicker | channels, personas, models, projects |
|
||||
| Editor | FileTree, CodeEditor, ChatPane (assist) | workspace, files, models |
|
||||
| Notes | NoteEditor, NoteGraph, ChatPane (assist) | notes, folders, graph |
|
||||
| Admin | AuditLog, SettingsForm, Table (various) | users, configs, models, health |
|
||||
| Settings | SettingsForm, KBPicker, PersonaPicker | profile, preferences, policies |
|
||||
|
||||
### Surface Registration
|
||||
|
||||
Today: hardcoded in Go (`pageEngine.RenderSurface("chat")`).
|
||||
|
||||
Future: manifest-driven. An admin-installed extension declares a
|
||||
surface in its manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"surfaces": [{
|
||||
"id": "triage-intake",
|
||||
"route": "/s/triage",
|
||||
"title": "Triage Intake",
|
||||
"components": ["chat-pane", "settings-form"],
|
||||
"data_requires": ["personas", "models"],
|
||||
"script": "surfaces/triage.js",
|
||||
"auth": "authenticated"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
The page engine reads registered surfaces, generates routes, assembles
|
||||
data loaders from the declared requirements, and renders a shell
|
||||
template that loads the surface's script.
|
||||
|
||||
**Route namespace:** Admin-created surfaces live under `/s/:slug` to
|
||||
avoid collision with core routes.
|
||||
|
||||
### Data Loaders
|
||||
|
||||
Each surface declares what data it needs. The page engine has a
|
||||
registry of data providers:
|
||||
|
||||
```
|
||||
"personas" → loads personas for the current user
|
||||
"models" → loads enabled models with health status
|
||||
"channels" → loads user's channels
|
||||
"workspace" → loads workspace by :wsId param
|
||||
"notes" → loads notes list
|
||||
...
|
||||
```
|
||||
|
||||
The surface's `data_requires` field pulls from this registry. Data is
|
||||
injected into the page as `window.__PAGE_DATA__` (existing pattern).
|
||||
The surface's JS reads from there on boot — no waterfall of API calls
|
||||
on page load.
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: Banners
|
||||
|
||||
The banner is global chrome outside the surface boundary. It exists
|
||||
at the top, bottom, or both — configured by the platform admin via
|
||||
`PUT /admin/settings/banner`.
|
||||
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"text": "DEVELOPMENT",
|
||||
"position": "both|top|bottom",
|
||||
"bg": "#007a33",
|
||||
"fg": "#ffffff"
|
||||
}
|
||||
```
|
||||
|
||||
The Go template base layout renders banners before and after the
|
||||
surface content area. CSS custom properties (`--banner-top-height`,
|
||||
`--banner-bottom-height`, `--banner-bg`, `--banner-fg`) allow surfaces
|
||||
to account for banner space without knowing banner state.
|
||||
|
||||
Banners are not configurable by extensions or themes. They are a
|
||||
platform-level trust signal.
|
||||
|
||||
---
|
||||
|
||||
## Themes
|
||||
|
||||
A theme is a set of CSS custom property overrides applied to `<html>`.
|
||||
|
||||
```css
|
||||
[data-theme="corporate"] {
|
||||
--bg: #f5f5f5;
|
||||
--text: #1a1a1a;
|
||||
--accent: #0066cc;
|
||||
--border: #d1d5db;
|
||||
--font-ui: 'Inter', sans-serif;
|
||||
--font-code: 'Fira Code', monospace;
|
||||
--radius: 4px;
|
||||
/* ... full property set */
|
||||
}
|
||||
```
|
||||
|
||||
### What a Theme Can Do
|
||||
|
||||
- Override any CSS custom property in the theme contract
|
||||
- Change colors, fonts, border radii, spacing scale
|
||||
- Switch between light and dark base palettes
|
||||
- Apply to every primitive, component, and surface uniformly
|
||||
|
||||
### What a Theme Cannot Do
|
||||
|
||||
- Add or remove DOM elements
|
||||
- Execute JavaScript
|
||||
- Modify component behavior or layout
|
||||
- Override banner appearance (platform chrome)
|
||||
- Access APIs or user data
|
||||
|
||||
### Theme Contract
|
||||
|
||||
The set of CSS custom properties that all primitives read from. This
|
||||
is the stable API between themes and the UI. Properties are namespaced:
|
||||
|
||||
```
|
||||
--bg, --bg-secondary, --bg-tertiary (backgrounds)
|
||||
--text, --text-secondary, --text-muted (typography)
|
||||
--accent, --accent-hover, --accent-muted (interactive)
|
||||
--border, --border-strong (edges)
|
||||
--success, --warning, --danger (semantic)
|
||||
--font-ui, --font-code (typefaces)
|
||||
--radius, --radius-lg (shapes)
|
||||
--shadow, --shadow-lg (elevation)
|
||||
--banner-bg, --banner-fg (read-only, set by platform)
|
||||
```
|
||||
|
||||
Themes are admin-installed. An admin uploads a CSS file that declares
|
||||
a `[data-theme="name"]` rule set. Users can select from installed
|
||||
themes in Settings → Appearance.
|
||||
|
||||
Built-in themes: `dark` (default), `light`. The system preference
|
||||
auto-detection (`prefers-color-scheme`) continues to work.
|
||||
|
||||
---
|
||||
|
||||
## Extension Hooks
|
||||
|
||||
Extensions interact with the application through a defined set of
|
||||
hooks. Each hook has a specific trigger point in the lifecycle and
|
||||
a clear contract for what the extension receives and returns.
|
||||
|
||||
### Hook 1: Pre-Completion
|
||||
|
||||
**When:** After the user sends a message, before the API request fires.
|
||||
**Receives:** The completion request object (model, messages, tools, etc.)
|
||||
**Returns:** Modified request object (or unmodified to pass through).
|
||||
**Use case:** Inject additional tools, modify system prompt, add context.
|
||||
|
||||
```js
|
||||
ctx.hooks.preCompletion(request => {
|
||||
request.tools.push(myCustomTool);
|
||||
return request;
|
||||
});
|
||||
```
|
||||
|
||||
### Hook 2: Stream Processing
|
||||
|
||||
**When:** On each SSE chunk during streaming.
|
||||
**Receives:** The chunk (content delta, tool_use, tool_result, etc.)
|
||||
**Returns:** Modified chunk (or null to suppress).
|
||||
**Use case:** Transform content, intercept tool calls, accumulate data.
|
||||
|
||||
```js
|
||||
ctx.hooks.streamChunk((chunk, context) => {
|
||||
if (chunk.type === 'tool_result' && chunk.name === 'image_gen') {
|
||||
context.displayContent.push({ type: 'image', src: chunk.data.url });
|
||||
return null; // suppress from text content
|
||||
}
|
||||
return chunk;
|
||||
});
|
||||
```
|
||||
|
||||
### Hook 3: Post-Render
|
||||
|
||||
**When:** After a message is rendered into the DOM.
|
||||
**Receives:** The message container element, the message object.
|
||||
**Returns:** Nothing (mutates DOM in place).
|
||||
**Use case:** Add action buttons, wrap elements, enhance display.
|
||||
|
||||
This is how extensions compose with each other's output. Extension A
|
||||
produces an image via display content. Extension B's post-render hook
|
||||
finds `<img>` elements and wraps them with action buttons. They don't
|
||||
know about each other — they agree on the DOM contract.
|
||||
|
||||
```js
|
||||
ctx.hooks.postRender((containerEl, message) => {
|
||||
containerEl.querySelectorAll('img[data-display-content]').forEach(img => {
|
||||
const actions = document.createElement('div');
|
||||
actions.className = 'image-actions';
|
||||
actions.innerHTML = '<button data-action="upscale">Upscale</button>';
|
||||
img.parentElement.appendChild(actions);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Hook 4: Display Content
|
||||
|
||||
**When:** After a completion finishes (all chunks received).
|
||||
**Receives:** The assistant message object, accumulated display content.
|
||||
**Returns:** Display content items to attach to the message.
|
||||
|
||||
Display content is **message-scoped, rendered inline, but excluded from
|
||||
the LLM context window**. It's not in `messages.content` and not sent
|
||||
back on the next turn.
|
||||
|
||||
```
|
||||
Message
|
||||
├── content (text — goes to LLM)
|
||||
├── attachments (files — go to LLM via multimodal assembly)
|
||||
└── display_content[] ← rendered inline, NOT sent to LLM
|
||||
├── { type: "image", src: "...", extension_id: "img-gen" }
|
||||
└── { type: "html", content: "<div>...", extension_id: "editor" }
|
||||
```
|
||||
|
||||
**Storage:** Display content is persisted in a `display_content` JSONB
|
||||
column on the message (or a junction table). It survives page reload
|
||||
and is included in `GET /channels/:id/path` responses.
|
||||
|
||||
### Hook 5: Tool Bridge
|
||||
|
||||
**When:** The LLM invokes a tool registered by the extension.
|
||||
**Receives:** Tool name and input parameters.
|
||||
**Returns:** Tool result (string or structured).
|
||||
|
||||
Existing mechanism — the WebSocket tool bridge from EXTENSIONS.md.
|
||||
Tool is registered via manifest, exposed to the LLM through the
|
||||
tools list, and executed client-side (browser tier) or server-side
|
||||
(starlark/sidecar tiers).
|
||||
|
||||
### Hook 6: Surface Injection
|
||||
|
||||
**When:** Surface boot (DOMContentLoaded).
|
||||
**Receives:** The surface ID and available mount points.
|
||||
**Returns:** Nothing (attaches to mount points).
|
||||
|
||||
Extensions declare which surfaces they target and which mount points
|
||||
they use:
|
||||
|
||||
```json
|
||||
{
|
||||
"surface_injections": [{
|
||||
"surface": "chat",
|
||||
"mount_point": "side-panel",
|
||||
"component": "my-panel.js"
|
||||
}, {
|
||||
"surface": "admin",
|
||||
"mount_point": "section",
|
||||
"section_id": "my-admin-section",
|
||||
"label": "Image Gen Settings"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Mount points are declared by each surface template:
|
||||
|
||||
```html
|
||||
<div data-mount="side-panel"></div>
|
||||
<div data-mount="section" data-section-id="..."></div>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Trust Model
|
||||
|
||||
Two tiers. The admin is the trust boundary.
|
||||
|
||||
### Privileged (Admin-Installed)
|
||||
|
||||
| Capability | Allowed |
|
||||
|-----------|---------|
|
||||
| Full DOM access | Yes |
|
||||
| Direct API calls (user's JWT) | Yes |
|
||||
| Modify other extensions' output (post-render) | Yes |
|
||||
| Create surfaces (own routes) | Yes |
|
||||
| Register tools | Yes |
|
||||
| Access all hook types | Yes |
|
||||
| Go template data loader | Yes |
|
||||
| Read `window.__PAGE_DATA__` | Yes |
|
||||
|
||||
The admin chose to install it. Same trust model as a VS Code extension
|
||||
or a WordPress plugin — you trust the publisher.
|
||||
|
||||
### Sandboxed (User-Installed, Future)
|
||||
|
||||
| Capability | Allowed |
|
||||
|-----------|---------|
|
||||
| Shadow DOM / iframe only | Yes |
|
||||
| API calls through message bridge | Yes (proxied, scoped) |
|
||||
| Modify other extensions' output | No |
|
||||
| Create surfaces | No |
|
||||
| Register tools | Limited (user-scoped) |
|
||||
| Access hooks | Post-render own output only |
|
||||
| Go template data loader | No |
|
||||
| Read `window.__PAGE_DATA__` | No |
|
||||
|
||||
Post-1.0 scope. Documented here to confirm the architecture
|
||||
accommodates it without redesign.
|
||||
|
||||
---
|
||||
|
||||
## Extension Lifecycle
|
||||
|
||||
### Boot Sequence
|
||||
|
||||
```
|
||||
1. Page loads (Go template renders surface shell + banner)
|
||||
2. Primitives initialize (CSS loaded, JS factories available)
|
||||
3. Components hydrate (attach to server-rendered DOM)
|
||||
4. Extensions.boot() — idempotent, runs on every surface
|
||||
a. Load extension manifests
|
||||
b. Register hooks (pre-completion, post-render, etc.)
|
||||
c. Register surface injections for current surface
|
||||
d. Initialize tool bridges
|
||||
5. Surface-specific JS runs (app.js for chat, editor-surface.js, etc.)
|
||||
```
|
||||
|
||||
`Extensions.boot()` replaces the current chat-specific
|
||||
`Extensions.loadAll()` / `Extensions.initAll()` pair and runs on
|
||||
every surface.
|
||||
|
||||
### Manifest Declaration
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "image-generator",
|
||||
"name": "AI Image Generator",
|
||||
"version": "1.0.0",
|
||||
"tier": "browser",
|
||||
"hooks": {
|
||||
"pre_completion": "hooks/pre-completion.js",
|
||||
"stream_chunk": "hooks/stream.js",
|
||||
"post_render": "hooks/post-render.js"
|
||||
},
|
||||
"tools": [{
|
||||
"name": "generate_image",
|
||||
"description": "Generate an image from a text description",
|
||||
"parameters": { ... }
|
||||
}],
|
||||
"surfaces": [{
|
||||
"id": "image-gallery",
|
||||
"route": "/s/gallery",
|
||||
"title": "Image Gallery",
|
||||
"components": ["chat-pane"],
|
||||
"data_requires": ["channels"],
|
||||
"script": "surfaces/gallery.js"
|
||||
}],
|
||||
"surface_injections": [{
|
||||
"surface": "chat",
|
||||
"mount_point": "message-actions",
|
||||
"script": "injections/image-actions.js"
|
||||
}],
|
||||
"theme": null,
|
||||
"settings_schema": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Display Content — Data Model
|
||||
|
||||
### Message Extension
|
||||
|
||||
```sql
|
||||
ALTER TABLE messages ADD COLUMN display_content JSONB DEFAULT '[]';
|
||||
```
|
||||
|
||||
Array of display items:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"type": "image",
|
||||
"src": "data:image/png;base64,...",
|
||||
"alt": "A cat in a spacesuit",
|
||||
"extension_id": "image-generator",
|
||||
"metadata": { "model": "dall-e-3", "size": "1024x1024" }
|
||||
},
|
||||
{
|
||||
"type": "html",
|
||||
"content": "<div class='chart'>...</div>",
|
||||
"extension_id": "data-viz",
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### API Contract
|
||||
|
||||
Display content is included in message objects returned by
|
||||
`GET /channels/:id/path` and `GET /channels/:id/messages`.
|
||||
|
||||
It is **excluded** from the message array sent to the LLM provider
|
||||
in `POST /chat/completions`. The completion handler strips
|
||||
`display_content` when assembling the provider request.
|
||||
|
||||
### Rendering
|
||||
|
||||
During `renderMessages()`, after the message content is rendered,
|
||||
each `display_content` item is rendered below the text content:
|
||||
|
||||
```html
|
||||
<div class="message-content">
|
||||
<p>Here's the image you requested:</p>
|
||||
</div>
|
||||
<div class="message-display-content">
|
||||
<img src="..." alt="..." data-display-content data-extension="image-generator">
|
||||
</div>
|
||||
```
|
||||
|
||||
The `data-display-content` attribute is the DOM contract that
|
||||
post-render hooks use to discover display content from any extension.
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
| Current State | Target State |
|
||||
|---------------|-------------|
|
||||
| `ui-primitives.js` (merged, with `esc()`, `createComponentRegistry()`, `componentMixin()`) | ✅ Done (v0.25.1) |
|
||||
| `ChatPane.create()` + 5 other components using shared mixin | ✅ Done (v0.25.1) |
|
||||
| 5 Go template routes | Surface registry (hardcoded core, manifest-driven extensions) |
|
||||
| `ctx.ui.replace()` / `ctx.ui.inject()` | Hook 6: Surface Injection with declared mount points |
|
||||
| `ctx.surfaces.register()` | Manifest `surfaces` field → page engine route registration |
|
||||
| Block renderers in `Extensions.initAll()` | `Extensions.boot()` on every surface |
|
||||
| Message content only (text + attachments) | `display_content` column for render-only visual content |
|
||||
| Block renderer pipeline (mermaid, katex) | Post-render hook + DOM contract (`data-display-content`) |
|
||||
| `[data-theme]` with CSS variables | Theme contract: stable CSS custom property namespace |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
This does not prescribe version numbers. It describes dependency order.
|
||||
|
||||
**Phase 1: Primitive consolidation** ✅ (v0.25.1)
|
||||
- `ui-primitives-additions.js` deleted, survivors merged into
|
||||
`ui-primitives.js`
|
||||
- `esc()` centralized (13 private copies eliminated)
|
||||
- Badge system unified (24 scattered classes → grouped in primitives.css)
|
||||
- CSS variable names normalized, duplicate selectors resolved
|
||||
- `showConfirm()` is the sole confirm pattern (bare `confirm()` purged)
|
||||
|
||||
**Phase 2: Component formalization** ✅ (v0.25.1)
|
||||
- `createComponentRegistry()` + `componentMixin()` extracted
|
||||
- 6 components (ChatPane, ModelSelector, UserMenu, CodeEditor, FileTree,
|
||||
NoteEditor) use the shared mixin. ~100 lines of boilerplate eliminated.
|
||||
- Each component has a Go template partial + JS hydration
|
||||
- `surface-nav.js` deleted; navigation is native in `ui-core.js`
|
||||
- Vendor scripts (`marked.min.js`, `purify.min.js`) and `ui-format.js`
|
||||
moved to `base.html` (loaded once, available to all surfaces)
|
||||
|
||||
**Phase 3: Extension hooks**
|
||||
- `Extensions.boot()` on every surface (v0.22.8)
|
||||
- Pre-completion and post-render hooks
|
||||
- Display content data model (message column + render pipeline)
|
||||
- Stream processing hook
|
||||
|
||||
**Phase 4: Surface registry**
|
||||
- Manifest-driven surface registration
|
||||
- Dynamic route generation in page engine
|
||||
- Data loader registry
|
||||
- Mount point declaration and injection
|
||||
|
||||
**Phase 5: Theme system**
|
||||
- Stable CSS custom property contract
|
||||
- Admin theme upload
|
||||
- User theme selection in Settings → Appearance
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Display content storage:** JSONB column on messages vs. separate
|
||||
`message_display_content` table. Column is simpler; table allows
|
||||
independent lifecycle (delete display content without touching
|
||||
message). Leaning column for KISS.
|
||||
|
||||
2. **Display content size limits:** Base64 images in JSONB could get
|
||||
large. May need blob storage for display content with URL references
|
||||
instead of inline data. Deferred until real usage patterns emerge.
|
||||
|
||||
3. **Extension ordering:** Post-render hooks from multiple extensions
|
||||
run in what order? Manifest priority field? Installation order?
|
||||
Alphabetical? Matters when Extension B adds buttons to Extension A's
|
||||
images — A must render first.
|
||||
|
||||
4. **Surface layout persistence:** If a surface supports resizable
|
||||
panes (editor: file tree width, chat pane width), where is that
|
||||
persisted? User settings? Per-workspace? Per-surface?
|
||||
|
||||
5. **Extension settings storage:** Extensions need per-user config
|
||||
(API keys for image gen, preferences for behavior). Current
|
||||
`UpdateUserExtensionSettings` endpoint exists. Is the schema
|
||||
sufficient or does it need structured validation from
|
||||
`settings_schema` in the manifest?
|
||||
Reference in New Issue
Block a user