switchboard/core
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Releases 9 Wiki Activity
This repository has been archived on 2026-04-03. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
113b98ace4eb6f1ddcb5d33e91d92d2fd8e0eb78
core/ROADMAP.md
xcaliber 113b98ace4 Changeset 0.13.1 (#65)
2026-02-25 23:56:27 +00:00

44 KiB
Raw Blame History

Roadmap — Chat Switchboard

See also:

  • ARCHITECTURE.md — Core services design, store layer, scope model
  • EXTENSIONS.md — Extension system spec (Browser/Starlark/Sidecar tiers, manifests, browser tool bridge, surfaces/modes, model roles)

Versioning (pre-1.0): 0.<major>.<minor> — hotfixes use quad: 0.x.y.z No compatibility guarantees before 1.0.

Dependency Graph

Features have real dependencies. This ordering respects them.

v0.9.x  Stability + Quick UX Wins              ✅
          │
v0.9.3  Content Visibility & Block Controls     ✅
          │
v0.9.4  API Key Encryption + Vault              ✅
          │
v0.10.0 Model Roles (utility + embedding)       ✅
        + Usage Tracking
          │
v0.10.2 Summarize & Continue                    ✅
          │
v0.10.3 Frontend Refactor                       ✅
          │
v0.10.4 Model Type Pipeline + Role Fixes        ✅
          │
v0.10.5 UI Primitives + Extension Surfaces      ✅
          │
v0.11.0 Extension Foundation (browser tier)     ✅
          │
  ┌───────┴──────────────┐
  │                      │
v0.12.0 File Handling  v0.13.0 Admin Panel
  + Vision ✅            Refactor (fullscreen)
  │                      │
  │                    v0.13.1 Web Search
  │                      + url_fetch
  │                      │
  └───────┬──────────────┘
          │
v0.14.0 Knowledge Bases (embedding role + file storage + pgvector)
          │
v0.15.0 Compaction (utility role + background job)
          │
v0.15.1 Context Recall Tools (attachment_recall, conversation_search)
          │
v0.16.0 Conversation Memory (cross-conversation user facts)
          │
v0.17.0 @mention Routing + Multi-model
          │
v0.18.0 Extension Surfaces + Modes
          │
v0.19.0 Smart Routing (policy on model roles)
          │
v0.20.0 Multi-Participant Channels + Presence
          │
v0.21.0 Auth Strategy (mTLS/OIDC) + Full RBAC
          │
  ┌───────┴──────────────┐
  │                      │
v0.22.0 Workflow       v0.23.0 Tasks /
  Engine                 Autonomous Agents
  (team-owned,           (service channels,
   staged processes,      scheduler, unattended
   human + AI collab)     execution)

✅ v0.9.0 — Schema Consolidation + BYOK

The "rewrite" release: 21 migrations collapsed to 1, store layer abstraction, and the model pipeline made real for end users.

Schema & Architecture

  • Consolidated schema (21 migrations → single 001_initial.sql)
  • Store layer abstraction (all DB via typed interfaces, no raw SQL in handlers)
  • Persona-as-trust-boundary model (replaces old presets concept)
  • Scope model (global/team/personal) for configs, personas, models
  • Backward-compatible API routes (v0.8 → v0.9 field aliases)

Capabilities System

  • Two-tier resolution: catalog (provider API, per-provider) → heuristic inference
  • Venice provider: full capability mapping incl. optimizedForCode
  • ResolveIntrinsic with gap-fill merging (catalog authoritative, heuristic fills gaps)
  • Preset capability inheritance via GetByModelIDAny fallback

BYOK (Bring Your Own Key)

  • Auto-fetch models on provider creation
  • User-facing refresh endpoint + "Refresh Models" button
  • Personal models in selector with scope=personal badge
  • Composite model IDs (configId:modelId) prevent cross-provider collisions

Bug Fixes (0.9.0.x)

  • API key storage (json:"-" tag silently dropped keys)
  • NULL model_default scan crash (sql.NullString)
  • Nil slice → JSON null (make([]T, 0))
  • Frontend error swallowing

Testing

  • Journey-based integration tests (API-only, 4-actor × 3-provider matrix)
  • Live Venice API integration test
  • Frontend JS test suite (107 tests, 27 suites)

✅ v0.9.1 — Capability Architecture + Docs

  • Removed static known model table (same model ≠ same capabilities across providers)
  • Resolution chain: catalog → heuristic only, no hardcoded table
  • Frontend KNOWN_MODELS removed — backend is sole source of truth
  • Heuristic patterns expanded (qwen3, gpt-5, grok, kimi, minimax, glm-5, gemma-3)
  • All providers updated to use InferCapabilities() directly
  • EXTENSIONS.md recovered and updated with Appendix A (Custom Renderers) and Appendix B (Model Roles)
  • ROADMAP.md, CHANGELOG.md, README.md comprehensive rewrite

✅ v0.9.2 — Quick UX Wins + Hardening

Polish that doesn't need new infrastructure. Ship fast, stabilize.

UX

  • Collapsible code blocks (long outputs get <details> wrapper)
  • HTML preview for generated content (sandboxed iframe toggle)
  • Token count estimate in input area (before send)
  • "Conversation is getting long" warning (context % thresholds)
  • New switchboard panel favicon (animated SVG + PNG + ICO)

Hardening

  • Proxy interception detection (Content-Type checks, actionable error messages)
  • Environment injection to frontend (window.__ENV__)
  • Enhanced debug diagnostics (NET:PROXY log type, Content-Type capture, env info)
  • Team admin audit scoping (see only their team's audit entries)

✅ v0.9.3 — Content Visibility & Block Controls

User control over what's expanded/collapsed in the message flow. Thinking blocks, tool calls, and code blocks all get consistent collapse controls instead of being hidden or auto-formatted without user agency.

Code Blocks

  • Collapse/expand toggle button on code block header (alongside Copy/Preview)
  • Auto-collapse at threshold (>15 lines), user can always toggle
  • Smooth max-height transition instead of <details> wrapping

Thinking Blocks

  • Always render thinking blocks (never hide from DOM)
  • showThinking setting controls default open/closed (default: false), not visibility
  • User can always click to expand regardless of setting
  • reasoning_content delta field support for models that stream thinking separately (Grok, etc.)
  • Reasoning accumulated during streaming, wrapped in <think> tags before persistence

Tool Calls

  • Persist tool call metadata in stored messages (tool_calls JSONB column)
  • Render tool calls on history load (not just during streaming)
  • Collapsed by default: "🔧 tool_name → done" with toggle for full I/O
  • Notes tool: "📝 View note" link that navigates to Notes panel
  • Tool result rendering fix (operator precedence bug showing "undefined results")

Notes Panel → Side Panel

  • Copy button on note cards (same pattern as code block copy)
  • Unified side panel replaces inline HTML preview and notes modal
  • Tabbed interface: Preview and Notes tabs
  • Desktop: slides in from right (480px), chat area shrinks via flexbox
  • Mobile: full-width overlay at z-index 200
  • Escape key integration (generation → command palette → side panel → modals)

Streaming Refactor

  • Extracted streamWithToolLoop() into stream_loop.go as single canonical streaming implementation
  • Both streamCompletion (normal chat) and Regenerate call shared function — only persistence differs
  • Regenerate handler now has full feature parity: tools, reasoning, tool_calls persistence
  • Fixed nil json.RawMessage causing PostgreSQL JSONB rejection on regen persistence

Regenerate Fix

  • Regen streams in-place (truncates display to parent before streaming)
  • No more "collapse to original branch" after completion

Favicon

  • Animated SVG favicon during generation (cascading opacity pulse on 4 dots)

Admin Scope

  • Admin models endpoint returns only global-scope models (not user BYOK or team models)
  • Bulk visibility toggle scoped to global providers only

✅ v0.9.4 — API Key Encryption + Vault

Two-tier encryption with a trust boundary between organizational keys (admin- accessible) and personal BYOK keys (user-only). Personal keys are protected by a per-user encryption key (UEK) that the platform admin cannot recover without the user's passphrase.

Threat Model

Threat Global/Team keys Personal BYOK keys
DB backup theft ✅ Protected (env-var AES) ✅ Protected (UEK AES)
SQL injection exfil ✅ Protected ✅ Protected
Admin reads DB directly ⚠️ Admin can decrypt (owns env var) ✅ Admin cannot decrypt (lacks user passphrase)
Admin modifies server code ⚠️ Out of scope (runtime access) ⚠️ Out of scope (runtime access)

Crypto Design

Two encryption paths, one mechanism (AES-256-GCM):

  1. Global/Team keys — encrypted with a key derived from ENCRYPTION_KEY env var. Admin-owned, admin-recoverable. Simple.

  2. Personal keys — encrypted with a per-user User Encryption Key (UEK). The UEK is a random 256-bit key generated at account creation, itself encrypted with a key derived from the user's password via Argon2id.

Password → Argon2id(salt) → Password-Derived Key (PDK)
PDK → AES-256-GCM-Decrypt(encrypted_uek) → UEK (plaintext, session-only)
UEK → AES-256-GCM-Decrypt(api_key_enc) → API key (plaintext, per-request)

The UEK lives in server memory for the duration of the session (alongside the JWT). It is never written to disk, logs, or the database in plaintext.

Schema Changes (migration)

-- Users: vault support
ALTER TABLE users ADD COLUMN encrypted_uek  BYTEA;
ALTER TABLE users ADD COLUMN uek_salt       BYTEA;    -- Argon2id salt
ALTER TABLE users ADD COLUMN uek_nonce      BYTEA;    -- GCM nonce for UEK wrap
ALTER TABLE users ADD COLUMN vault_set      BOOLEAN NOT NULL DEFAULT false;

-- API configs: encrypted key storage
ALTER TABLE api_configs ADD COLUMN api_key_enc  BYTEA;
ALTER TABLE api_configs ADD COLUMN key_nonce    BYTEA;
ALTER TABLE api_configs ADD COLUMN key_scope    TEXT NOT NULL DEFAULT 'global';
-- key_scope: 'global' | 'team' | 'personal'
-- global/team: decrypt with env-var-derived key
-- personal:    decrypt with user's UEK

-- Team providers: same pattern
ALTER TABLE team_providers ADD COLUMN api_key_enc  BYTEA;
ALTER TABLE team_providers ADD COLUMN key_nonce    BYTEA;

Migration backfills: encrypt existing plaintext keys using the env var key (all existing keys are global/team scope). Drop plaintext api_key column after backfill. If ENCRYPTION_KEY is not set, migration refuses to run (fail-safe).

Auth Mode Integration

Builtin auth — password serves double duty: authentication + UEK derivation. No additional UX. On login, derive PDK → unwrap UEK → cache in session. User never knows the vault exists.

mTLS / OIDC (v0.21.0) — authentication is external (cert/token). Vault passphrase is conditionally required:

External auth → auto-authenticated
  ├── Personal providers DISABLED → straight to app, no prompt
  └── Personal providers ENABLED
        ├── First visit  → "Set a vault passphrase to protect your API keys"
        └── Returning    → "Unlock your vault" (passphrase field, pre-filled username)

Terminology: "vault passphrase" for mTLS/OIDC users (distinct from their non-existent account password). Builtin auth users never see this term.

Backend Implementation

server/crypto/vault.go — pure encryption/decryption, no DB awareness:

  • DeriveKey(password, salt) → key (Argon2id, 64MB memory, 3 iterations)
  • GenerateUEK() → (uek, error) (crypto/rand, 32 bytes)
  • WrapUEK(uek, pdk) → (ciphertext, nonce, error)
  • UnwrapUEK(ciphertext, nonce, pdk) → (uek, error)
  • Encrypt(plaintext, key) → (ciphertext, nonce, error)
  • Decrypt(ciphertext, nonce, key) → (plaintext, error)
  • DeriveEnvKey(envVar) → key (HKDF-SHA256 with domain separation context)

Session UEK caching — UEK stored in a sync.Map keyed by user ID, evicted on logout / token expiry. Completion handler retrieves UEK from cache to decrypt personal provider keys at request time.

Provider resolution — key_scope determines decryption path:

func (s *Store) DecryptAPIKey(config ApiConfig, uekCache *sync.Map) (string, error) {
    switch config.KeyScope {
    case "global", "team":
        return crypto.Decrypt(config.APIKeyEnc, config.KeyNonce, envDerivedKey)
    case "personal":
        uek, ok := uekCache.Load(config.UserID)
        if !ok { return "", ErrVaultLocked }
        return crypto.Decrypt(config.APIKeyEnc, config.KeyNonce, uek.([]byte))
    }
}

Edge Cases

Scenario Behavior
User changes password (builtin) Re-derive PDK, re-encrypt UEK with new PDK. Personal API keys unchanged (keyed to UEK, not password).
Admin resets user password Old UEK unrecoverable. Personal keys destroyed. Admin UI warns before confirming.
mTLS user forgets passphrase Self-service reset: destroys encrypted personal keys. UX: "This will erase your stored API keys."
Admin disables personal providers Keys stay encrypted in DB (inert). Re-enable later → keys accessible if user remembers passphrase.
ENCRYPTION_KEY not set Startup refuses to run if encrypted keys exist in DB. Fresh install: keys stored plaintext with deprecation warning in logs + admin panel.
ENCRYPTION_KEY rotated Admin CLI tool: switchboard vault rekey — re-encrypts all global/team keys with new env key. Personal keys unaffected (keyed to UEK).

Frontend Changes

  • Provider management (Settings → Providers): no change to UX for builtin auth
  • mTLS/OIDC splash: conditional vault passphrase prompt (new component)
  • Admin → Users → Reset Password: warning dialog about personal key destruction
  • Admin → Settings: indicator for ENCRYPTION_KEY status (set / not set)

Checklist

Crypto core

  • server/crypto/vault.go — Argon2id derivation, AES-256-GCM wrap/unwrap
  • server/crypto/vault_test.go — round-trip, wrong-password, corruption (11 tests)
  • Env-var key derivation + validation on startup (HKDF-SHA256 with domain separation)
  • server/crypto/cache.go — UEK cache with copy-on-read/write, memory zeroing
  • server/crypto/resolver.go — KeyResolver routes decrypt/encrypt by key_scope
  • server/crypto/backfill.go — startup migration for plaintext → encrypted keys

Schema + migration

  • Migration 003_vault.sql: encrypted columns, key_scope backfill
  • ENCRYPTION_KEY enforcement (refuse to start if keys exist but no env var)
  • Unencrypted fallback when ENCRYPTION_KEY not set (raw bytes, graceful degradation)

Auth integration

  • UEK generation on user creation (builtin auth)
  • UEK unwrap on login, cache in session
  • UEK eviction on logout (memory zeroing)
  • UEK re-wrap on password change (deferred — Settings handler)
  • UEK destruction on admin password reset with confirmation (deferred)
  • Vault passphrase prompt for mTLS/OIDC (deferred — v0.21.0 dependency)

Provider key lifecycle

  • Encrypt on provider create/update (all 6 write paths: admin global, personal, team)
  • Decrypt on completion request (all 4 read paths: completion, admin fetch, personal fetch, team fetch)
  • ErrVaultLocked handling (prompt user to unlock)

Admin tooling

  • switchboard vault rekey CLI command (v0.12.0)
  • switchboard vault status CLI command (v0.12.0)
  • Admin UI: encryption status indicator (v0.12.0 — Settings tab)
  • Admin UI: password reset warning (v0.10.0 — vault destruction dialog)

CI/CD

  • ENCRYPTION_KEY Gitea secret → k8s switchboard-encryption secret sync
  • Backend k8s manifest: optional secretKeyRef for ENCRYPTION_KEY
  • Pipeline version bump to v0.9.4

Security fix (bonus)

  • DOMPurify: strict ALLOWED_TAGS allowlist (was permissive ADD_TAGS default)
  • Think-block placeholder \n\n padding (prevents fence fusion with </think>)
  • Unclosed code fence auto-close before marked.parse (streaming protection)

✅ v0.10.0 — Model Roles + Usage Tracking + Vault Debt

Three tracks of infrastructure that everything downstream depends on.

Vault Debt (Security Fixes)

  • UEK re-wrap on user password change (prevents permanent key loss)
  • UEK destruction on admin password reset (vault + personal BYOK keys deleted)
  • Admin UI: password reset confirmation with vault destruction warning
  • Audit logging for vault destruction events

Model Roles (see EXTENSIONS.md Appendix B)

  • model_roles in global_settings: named slots (utility, embedding, generation)
  • Each role: primary provider+model, optional fallback provider+model
  • Team-level role overrides (team.settings JSONB)
  • Provider.Embed() interface — OpenAI/Venice/OpenRouter implement, Anthropic returns ErrNotSupported
  • roles.Resolver package: Complete() + Embed() with automatic fallback
  • Admin API: GET/PUT /admin/roles/:role, POST /admin/roles/:role/test
  • Team API: GET/PUT/DELETE /teams/:id/roles/:role
  • Admin UI: role configuration tab (select provider + model per role, test fire)

Usage Tracking

  • Streaming token capture fix — OpenAI: stream_options.include_usage + parse usage chunk; Anthropic: parse message_start/message_delta usage events
  • Cache token fields across all types (CompletionResponse, StreamEvent, streamResult)
  • usage_log table: channel_id, user_id, model_id, provider_config_id, role, token counts (prompt, completion, cache_creation, cache_read), cost, timestamp
  • model_pricing table: provider+model, input/output/cache per-M, source (catalog/manual)
  • Cost calculated at insert time (baked, not query-time)
  • BYOK exclusion: admin views filter provider_scope != 'personal', BYOK gets separate endpoint
  • Model pricing from catalog sync (won't overwrite manual overrides)
  • Admin API: GET /admin/usage, GET /admin/usage/users/:id, GET /admin/usage/teams/:id
  • Admin API: GET/PUT/DELETE /admin/pricing
  • User API: GET /usage (personal usage summary)
  • Admin UI: usage dashboard (period selector, group-by, totals cards, breakdown table, pricing table)
  • Team admin UI: usage dashboard for team-owned providers (GET /teams/:teamId/usage)
  • Token accumulation across multi-tool-call iterations in stream_loop.go

Schema (migration 004)

  • usage_log with indexes on user, created_at, provider, model, scope
  • model_pricing with unique constraint on (provider_config_id, model_id)
  • model_roles seed in global_settings

✅ v0.10.1 — Polish + UX

Spit and polish release: admin system prompt injection, team management modal extraction, persona tab, side panel enhancements, code download, and critical OpenAI streaming usage race fix.

  • Admin system prompt (global, injected before user/preset prompts)
  • Personas tab (moved from Models, policy-gated)
  • Team Management modal (extracted from Settings, tabbed, multi-team picker)
  • Preview pane: clear button, fullscreen toggle, resizable
  • Code block: download button (infers extension from language tag)
  • Personal Usage scoped to BYOK only
  • OpenAI streaming usage race fix (deferred Done event until usage chunk)
  • Usage logging zero-token guard removal

v0.10.2 — Summarize & Continue + Role Cleanup ✅

First consumer of the utility model role. User-triggered conversation compaction — not automatic (that's v0.15.0).

Depends on: utility model role (v0.10.0).

Summarize & Continue

  • "Summarize and continue" button in context warning bar (≥75% context)
  • POST /channels/:id/summarize — calls utility role with conversation history
  • Summary inserted as tree node with metadata.type = "summary" + boundary metadata
  • loadConversation recognizes summary boundaries: replaces pre-summary messages with summary system message
  • Multiple summaries stack (re-summarize after context fills again)
  • Fork-aware: summary is a tree node, branches have independent boundaries
  • Original messages preserved: "show full history" toggle reveals collapsed messages
  • Usage logged against utility role with cost tracking

BYOK Role Overrides

  • Personal role resolution chain: personal → team → global
  • User Settings → Model Roles tab (BYOK providers only)
  • Stored in user.settings JSONB (model_roles key, same shape as team overrides)
  • Only utility and embedding roles (generation removed)

Generation Role Removal

  • RoleGeneration removed from ValidRoles (was "generation" for image/media)
  • Removed from admin Roles tab UI
  • Image gen will be extension-managed (v0.11.0), not a role slot

Utility Rate Limiting

  • utility_rate_limit global setting (default: 20 calls/hour/user, 0 = unlimited)
  • Check against usage_log before executing org-funded utility calls
  • BYOK calls exempt (user's own key, user's own cost)
  • 429 response with clear message when exceeded

v0.10.3 — Frontend Refactor ✅

Zero features. Split the two monolith frontend files (app.js 2,940 lines, ui.js 2,582 lines) into 13 domain-scoped files averaging ~544 lines each. Vanilla JS, no modules, no build step, no function renaming.

See REFACTOR-0.10.3.md for full plan.

  • Extract ui-format.js — markdown, code blocks, esc(), helpers
  • Extract ui-settings.js — settings tabs, teams, providers, user prefs
  • Extract ui-admin.js — admin modal tabs, all admin rendering
  • Extract tokens.js — context tracking, token estimation
  • Extract notes.js — notes panel, editor, multi-select
  • Extract chat.js — chat ops, send, regen, edit, branch, summarize
  • Extract settings-handlers.js — settings save, provider CRUD, cmd palette
  • Extract admin-handlers.js — admin actions, presets, team management
  • Slim app.js to orchestrator — state, init, boot, auth, listener dispatch
  • Update sw.js SHELL_FILES, index.html script tags
  • All 159+ frontend tests pass, node --check on all files

v0.10.4 — Model Type Pipeline + Role Fixes ✅

Provider APIs report model types (chat, embedding, image) — now captured end-to-end through sync → catalog → resolver → frontend. Role dropdowns filter models by type instead of showing everything.

  • Migration 005_model_type.sql — model_catalog.model_type column
  • providers.Model.Type — captured from provider wire response
  • Venice: reads type field, normalizes (text→chat, embedding→embedding)
  • OpenAI: wire struct extended with optional type for compatible APIs
  • CatalogSyncEntry.ModelType → CatalogEntry.ModelType → UserModel.ModelType
  • Frontend model_type mapped through fetchModels()
  • Admin role dropdowns: filter by model_type matching role
  • User role dropdowns: filter by model_type matching role
  • adminSaveRole() reloads UI after save (was showing "Saved" without refresh)

v0.10.5 — UI Primitives + Extension Surfaces ✅

Consolidate duplicated UI patterns into shared primitives with extension-ready interfaces. Pre-extension infrastructure: the primitives we build now become the ctx.* extension API surface in v0.11.0 with no rewrite needed.

  • Providers registry — single source of truth for types, labels, endpoints
  • Roles registry — names, type filters, hints
  • renderCapBadges() — consolidates 3 badge builders (compact + detailed modes)
  • renderProviderForm() — replaces 3 form implementations, dual-mode create/edit
  • renderProviderList() — replaces 3 list renderers, event delegation
  • renderRoleConfig() — replaces 2 role UIs + 4 handlers
  • renderUsageDashboard() — replaces 3 usage renderers (compact + full)
  • Admin provider form gets endpoint auto-fill (was missing)
  • Team provider form consolidated from 2 separate forms to 1 dual-mode
  • Removed ~360 lines of duplicated code + ~40 lines of static HTML
  • showConfirm() — styled confirm dialog replacing all 16 native confirm() calls
  • .popup-menu shared CSS + createPopupMenu() primitive for consistent menus
  • Sidebar collapse icon always visible (dimmed), brightens on hover

✅ v0.11.0 — Extension Foundation (Browser Tier)

The platform play. See EXTENSIONS.md for full spec.

Core Loader (EXTENSIONS.md §4, §7)

  • extensions.js — loader, registry, scoped context, renderer pipeline (~400 lines)
  • Extensions.register() API with permission-aware scoped contexts
  • Manifest format: _script inline or entry file, permissions, settings schema
  • extensions + extension_user_settings tables (migration 006)
  • Admin endpoints: CRUD, enable/disable, asset serving
  • /api/v1/extensions/:id/assets/*path — public (no auth, <script> tag compatible)
  • Auto-seeder: scans extensions/builtin/*/ on startup, upserts manifests
  • Admin Extensions tab with Edit button (view/modify manifest + script inline)

Browser Tool Bridge (EXTENSIONS.md §5)

  • ctx.tools.handle() API for browser tool registration
  • Tool schema collection: merge server + extension tools in completions
  • tool.call.* / tool.result.* WebSocket routing via EventBus hub
  • 30-second timeout with error recovery
  • Events.waitFor() pattern for request/response tool calls

Custom Renderer API (EXTENSIONS.md Appendix A)

  • ctx.renderers.register() — block, post, and inline types with pattern/match
  • Renderer pipeline: block renderers intercept code fences, post renderers process DOM
  • Case-insensitive language matching (LLMs capitalize fence tags)
  • Nested ```markdown fence unwrapping (LLMs wrap responses in markdown fences)
  • Extension-owned styles via _injectStyles() / destroy() lifecycle

Server Tools (auto-registered via init())

  • datetime — current date/time/timezone/day/unix/ISO-week (models hallucinate dates)
  • calculator — recursive-descent evaluator: arithmetic, 17 functions, constants

Built-in Browser Extensions (6 total, self-contained with own CSS)

  • Mermaid — block + post renderer, SVG diagrams, dark mode, local vendor + CDN fallback
  • KaTeX — block renderer (```latex/math/tex ```) + post renderer (inline$.../...$`)
  • CSV Table — block renderer, RFC 4180 parser, sortable columns, numeric-aware sorting
  • Diff Viewer — block renderer, red/green syntax highlighting, stats badge
  • JS Sandbox — browser tool (js_eval), sandboxed iframe, 10s timeout, console capture
  • Regex Tester — browser tool (regex_test), multi-input matching, named groups

Bugfixes

  • WebSocket token field mismatch (tokens.access → tokens.accessToken)
  • Extension asset 401 (moved asset route to public group — <script> tags can't send auth headers)
  • runExtensionPostRender is not defined (stale SW cache, added typeof guard)
  • Extension init order (moved loadAll()/initAll() before loadChats())
  • K8s Ingress WebSocket routing (pathType: Exact → Prefix for Traefik longest-prefix-wins)
  • Notes missing post-render call (mermaid/KaTeX wouldn't render in note view)

Diagnostics

  • Test 5: Browser extensions (loaded count, active/inactive, renderers, tool handlers)
  • Test 6: Service Worker cache (registration, scope, state, cache names, entry counts)
  • Purge Cache button in Debug Log modal (deletes all SW caches, unregisters workers)

v0.12.0 — File Handling + Vision

File input into chat — table stakes for serious use.

Storage Backend

  • S3-compatible API (MinIO, Ceph RGW, AWS — anything with S3 API)
  • Local PVC backend (single-node / dev)
  • Admin config: backend health status, orphan cleanup panel
  • Reused by KBs (v0.14.0) and compaction snapshots (v0.15.0)

Chat Integration

  • attachments table (metadata in PG, blobs in object store)
  • Image/file upload in chat messages (📎 button, drag-and-drop)
  • Multimodal message assembly for vision-capable models
  • Text extraction pipeline (PDF, DOCX, TXT — inline + sidecar modes)
  • Paste-to-upload (clipboard images + large text auto-attach)
  • Staged attachment strip with extraction status polling
  • Auth-aware rendering (blob URLs with Bearer token)
  • Image lightbox viewer
  • Vision capability hints on image attachments
  • Document generation output storage (extension → attachment → download)

v0.13.0 — Admin Panel Refactor

The admin modal has grown to 12 tabs in a horizontal overflow strip. Adding search provider config, KB management, and workflow settings would make it worse. Replace the modal with a fullscreen admin panel.

Depends on: nothing (frontend-only, no API changes).

Layout (modeled after Open WebUI's admin panel):

  • Fullscreen overlay (or /admin route) replaces the modal
  • Top bar: 4 category tabs (horizontal)
  • Left sidebar: section tabs within the active category
  • Main content: full viewport width for the active section

Category → Section mapping:

Category Sections Current tabs absorbed
People Users, Teams Users, Teams
AI Providers, Models, Personas, Roles Providers, Models, Presets, Roles
System Settings, Storage, Extensions Settings, Storage, Extensions
Monitoring Usage, Audit, Stats Usage, Audit, Stats
  • Fullscreen admin layout: top categories + left sidebar sections
  • Migrate all 12 existing tab contents into new layout (no API changes)
  • Category → section routing with URL hash or state (e.g. #admin/ai/models)
  • Responsive: sidebar collapses on narrow viewports
  • Keyboard shortcut: Esc returns to chat
  • Admin button in sidebar opens fullscreen panel (replaces modal trigger)
  • Remove old adminModal HTML + CSS
  • Preserve all existing loadAdmin*() functions — new layout calls same loaders
  • Roles moved from People → AI category
  • Presets → Personas label rename (UI only, API/data unchanged)
  • Banner-aware: admin panel respects --banner-top/bottom-height
  • CSS design token cleanup: ghost variables, hardcoded colors → tokens
  • Global form element theming (bare selects/inputs inherit theme)
  • Zoom scaling: admin panel + side panel respect appearance slider
  • CI fix: TruncateAll() includes platform_policies + global_config

v0.13.1 — Web Search + URL Fetch

Built-in tools that extend the tool framework shipped in v0.11.0. Admin configuration lives in the new admin panel (System → Settings or AI → Providers).

Depends on: admin panel refactor (v0.13.0) for config UI, tool framework (v0.11.0).

Search provider abstraction:

  • SearchProvider interface: Search(ctx, query, maxResults) → []SearchResult
  • DuckDuckGo provider (no API key required — default, works out of the box)
  • SearXNG provider (self-hosted, fits airgapped story, configurable endpoint)
  • Admin config: select active provider, set endpoint/API key, result count
  • Provider config stored in global_config as search_config key
  • Runtime apply on save (no restart required) + load on startup

Tools:

  • web_search tool: calls active search provider, returns title + snippet + URL
  • url_fetch tool: HTTP GET + content extraction (HTML→text, SSRF protection)
  • Results injected into context as tool results (existing tool framework handles this)
  • Rate limiting / concurrency control per-provider

Tool categories + toggle:

  • ToolDef.Category field: search, notes, utilities, browser
  • disabled_tools[] per-request filtering (completion + regen + edit paths)
  • GET /api/v1/tools endpoint: returns [{name, description, category}]
  • Chat-bar tools toggle popup: category-level enable/disable
  • localStorage persistence of disabled state
  • ToolDef.DisplayName field: human-readable name for UI (e.g. "Search" not "web_search")
  • Sub-menu per category: expand-in-place to show individual tools
  • Tri-state category checkbox: all on / all off / indeterminate (partial)
  • Per-tool toggle within categories (child checkboxes)

Future providers (abstraction supports but not in v0.13.1):

  • Brave Search (API key), Tavily, Kagi, SerpAPI, Google PSE

v0.14.0 — Knowledge Bases

Depends on: embedding model role (v0.10.0), file storage (v0.12.0).

  • Embedding pipeline: chunking (recursive, semantic), generation via embedding role
  • pgvector storage and similarity search
  • knowledge_bases table with team scoping
  • KB document storage via v0.12.0 storage backend
  • kb_search tool (uses existing tool framework)
  • Context injection in completion flow
  • Notes embedded too (once pipeline exists)
  • Per-channel KB toggle

v0.15.0 — Compaction

Depends on: utility model role (v0.10.0).

  • Auto-compaction service: background job that calls utility model to summarize
  • Channel-scoped: triggers when context exceeds threshold
  • Summaries stored as system messages
  • Configurable: per-channel opt-in/out, summary model override
  • Admin controls for resource limits

v0.15.1 — Context Recall Tools

Depends on: file handling (v0.12.0), tool framework (v0.11.0).

Attachments are injected into context once (at send time) and are not replayed on subsequent turns — the model's own response acts as a natural summary. After compaction (v0.15.0) or in long conversations, the model may need to re-read source material. These tools bridge that gap.

attachment_recall tool

  • attachment_recall(attachment_id) — re-reads attachment content from storage
  • Images: re-inject as base64 content part (vision gating still applies)
  • Documents: return extracted text
  • Security: scoped to current channel's attachments only
  • Extend ExecutionContext with optional storage backend reference
  • Tool definition includes attachment list hint so model knows what's available

conversation_search tool

  • Keyword search over compacted/archived messages in current channel
  • Returns matching message excerpts with timestamps
  • Useful post-compaction when summary dropped details the user asks about

Token estimator improvements

  • Include attachment token estimates in context counter
  • Images: use provider-specific sizing rules (e.g. tile-based for OpenAI)
  • Documents: count extracted text tokens
  • Staged attachments reflected in input token counter before send

v0.16.0 — Conversation Memory

Depends on: compaction (v0.15.0), knowledge bases (v0.14.0).

Long-term memory across conversations. Unlike KB (static documents) or compaction (within-conversation), this captures user-level facts and preferences that persist across channels.

  • user_memory table: user_id, key, value, source_channel_id, confidence, created_at, updated_at
  • memory_save tool: LLM can explicitly store a fact ("user prefers Python", "project deadline is March 15")
  • memory_recall tool: LLM queries stored facts relevant to current context
  • Auto-extraction: background job identifies memorable facts from conversations (opt-in)
  • Memory injection: relevant facts injected into system prompt at completion time
  • User controls: view, edit, delete memories in Settings
  • Admin controls: enable/disable, retention policies, per-team scoping
  • Privacy: memories never cross team boundaries

v0.17.0 — @mention Routing + Multi-model

The channel schema already supports multiple models. This adds routing.

  • @mention parsing in messages (users and AI models)
  • Resolve mentions against channel_models
  • Multi-model channels: fire completions per mentioned model
  • "Add a model" UI per channel
  • Enables: second opinions, cross-model conversations

v0.18.0 — Extension Surfaces + Modes

Depends on: extension foundation (v0.11.0). See EXTENSIONS.md §6.

  • Surface registration and activation API
  • Mode selector in sidebar (below brand, above chat history)
  • ctx.ui.replace() / ctx.ui.restore() for region management
  • surface.activated / surface.deactivated events
  • Editor mode (extension: file tree + code editor + AI chat split pane)
  • Article mode (extension: outline + rich text + AI assistant panel)

v0.19.0 — Smart Routing

Depends on: model roles (v0.10.0), usage tracking (v0.10.0). Rules-based routing engine — policy, not ML.

  • Admin routing policies: "cheapest model with required capabilities", "prefer private providers, fallback to cloud", "for team X, use Y"
  • Fallback chains: primary provider down → next with same model
  • Cost-aware: factor pricing into routing decisions
  • Latency-aware: track response times per provider, prefer faster

v0.20.0 — Multi-Participant Channels + Presence

The channel foundation for workflows and real-time collaboration. Today channels are single-user direct chats. This release makes channels a shared space that multiple actors can inhabit.

Depends on: extension surfaces (v0.18.0), WebSocket infrastructure (already exists).

Channel Participants

  • channel_participants table: channel_id, participant_type (user, session, persona), participant_id, role (owner, member, observer), joined_at
  • Channel access queries rewritten: WHERE user_id = $1 → participant-based lookup
  • Backward compatible: existing single-user channels auto-have one participant (owner)
  • Channel type column: direct (current), group (multi-user), workflow (staged)

Presence + Real-time

  • Typing indicators and presence (who's online, who's in this channel)
  • Participant join/leave events via WebSocket
  • Co-editing for extension surfaces (editor mode, article mode)
  • Cursor sharing, selection highlighting

Channel-Scoped Notes

  • Optional channel_id FK on notes (NULL = personal notebook, set = channel-attached)
  • Notes created by tool calls within a channel auto-attach to that channel
  • Channel notes visible to all participants (not just creator)

v0.21.0 — Auth Strategy (mTLS/OIDC) + Full RBAC

Enterprise auth modes on top of the teams foundation. Depends on: vault passphrase support from v0.9.3 (conditional unlock for personal providers under external auth).

  • AUTH_MODE env var: builtin | mtls | oidc
  • All three resolve to the same internal user model
  • auth_source + external_id columns on users table
  • mTLS: header trust, auto-provision from cert DN
  • OIDC: Keycloak/Okta token validation, claim extraction, role mapping
  • Per-source auto-activate policy: auto_activate, default_team, default_role
  • Vault passphrase prompt for mTLS/OIDC users (v0.9.3 conditional unlock)
  • Fine-grained permissions: model access, KB write, task create, token budgets

Anonymous / Session Participants

  • Session-based identity for unauthenticated visitors (workflow intake)
  • mTLS: cert CN/fingerprint as stable session identity (no users row required)
  • Non-mTLS: ephemeral session token (cookie or URL token)
  • Session participants can send messages and trigger tool calls within workflow channels
  • Session identity visible to assigned team members (cert DN, display name, or "Anonymous #N")

v0.22.0 — Workflow Engine

Team-owned, stage-based process execution. The channel is the runtime, personas drive each stage, and the existing tool/notes infrastructure handles structured data collection. See ARCHITECTURE.md — Workflow Architecture for the conceptual model.

Depends on: multi-participant channels (v0.20.0), anonymous identity (v0.21.0).

Workflow Definitions

  • workflows table: team_id (owner), name, description, is_active, entry_mode (public_link, team_internal, api)
  • workflow_stages table: workflow_id, ordinal, persona_id, assignment_team_id, form_template (JSONB, structured note schema), transition_rules (JSONB)
  • Team-admin CRUD: create/edit/delete workflows and stages
  • Workflow versioning: edits create new version, active instances continue on their version

Workflow Instances (Channels)

  • Workflow-typed channels: type='workflow', workflow_id, current_stage, stage_version
  • Entry point: public link generates workflow channel, adds anonymous participant, binds stage 1 persona
  • Stage transitions: triggered by AI (tool call), team member (button), or rule (form complete)
  • On transition: swap active persona, notify assignment team, update channel metadata

AI Intake

  • Stage persona drives structured collection via system prompt
  • Form template → persona system prompt injection ("collect these fields: ...")
  • Tool calls create channel-scoped notes as form responses
  • AI determines readiness: "I have everything needed" → trigger transition

Assignment + Queue

  • Assignment queue: unassigned workflow channels visible to team members
  • Claim model: team member claims channel (becomes participant with member role)
  • Round-robin / rule-based auto-assignment (optional, per-stage config)
  • Assignment notifications via WebSocket + optional webhook

Team Member Collaboration

  • Assigned member sees full history (AI intake + artifacts + notes)
  • Persona remains active — assists both visitor and team member
  • Member can trigger stage transitions, add notes, invoke tools
  • Member can reassign to different team member or escalate to different team

Frontend

  • Workflow builder UI in team admin panel (stage list, persona picker, form template editor)
  • Queue view: sidebar section showing unassigned workflow channels for user's teams
  • Channel header: workflow stage indicator, transition controls
  • Anonymous visitor view: minimal UI, persona-driven conversation (public link entry)

v0.23.0 — Tasks / Autonomous Agents

Unattended execution — workflows without a human in the loop. Depends on: workflow engine (v0.22.0).

  • Scheduler + task runner (cron-like triggers for workflow instantiation)
  • task_create tool (AI can spawn sub-workflows)
  • type: 'service' channels: workflow instances with no human participants
  • Execution budgets: max tokens, max tool calls, max wall-clock per stage
  • Admin controls for resource limits and kill switches
  • Completion webhooks (notify external systems when workflow completes)

TBD (unscheduled — real features, no immediate need)

Items that are real but don't yet have a version assignment. Pull left based on need.

Extension System — Additional Extensions

  • Image generation tool (browser or sidecar, provider-agnostic)
  • STT/TTS (browser extension, Web Speech API — personal use case)
  • Code execution sandbox (server-side, container isolation)

Extension System — Server Tiers

  • Starlark runtime integration (Tier 1 — server sandbox)
  • Sidecar HTTP tool protocol (Tier 2 — container isolation)
  • Server-side tool execution in completion handler

Desktop + Mobile

  • Desktop app (Tauri)
  • Full PWA with offline capability
  • Mobile-optimized layouts (beyond current responsive)

Data + Portability

  • Bulk export/import (account data, conversations, settings)
  • ChatGPT/other tool import
  • GDPR-style "download my data"
  • Backup/restore CronJob manifests

UX / Multi-Seat

  • Per-chat model/preset persistence (server-side) ✅ (v0.12.0 — channels.settings.last_selector_id JSONB merge, localStorage write-through cache)

Platform

  • Rate limiting per user/team/tier (token budgets)
  • Provider health monitoring + key rotation
  • Multi-tenant SaaS mode
  • Plugin/extension marketplace
  • Virtual scroll for long conversations
  • SQLite backend option (single-user / dev)
Reference in New Issue View Git Blame Copy Permalink