step 5 (complete): docs purge, fresh ARCHITECTURE + ROADMAP + CHANGELOG
Purged 29,966 lines of stale chat-switchboard documentation: - 41 docs/ files (ICD specs, design docs, archive, workflow docs) - 5 root MD files (CHANGESET, TURNOVER, ICD-DRIFT-AUDIT, etc.) New documentation: - docs/ARCHITECTURE.md — kernel components, design principles, data layer - ROADMAP.md — v0.1.0 through v0.5.0 MVP with decision log - CHANGELOG.md — fresh, starting from v0.1.0 fork - README.md — rewritten for switchboard-core Also in this commit: - config.go: stripped 7 dropped fields, DB default → switchboard_core - pages/loaders.go: stripped provider/model/notes/projects loaders - pages/pages.go: stripped persona store lookup - handlers/workflows.go: stripped persona tool grants - main.go: stripped team provider routes, avatar routes Production code: zero references to deleted packages, stores, or models. -29,966/+352 lines across 73 files.
This commit is contained in:
@@ -1,590 +1,176 @@
|
||||
# Architecture — Chat Switchboard v0.30.2
|
||||
# Switchboard Core — Architecture
|
||||
|
||||
## Deployment Modes
|
||||
|
||||
Three Docker images support different deployment scenarios:
|
||||
|
||||
| Image | Dockerfile | Contents | Use Case |
|
||||
|-------|-----------|----------|----------|
|
||||
| **Unified** | `Dockerfile` | nginx + Go backend | Dev, docker-compose, single-node |
|
||||
| **Backend** | `server/Dockerfile` | Go binary only | K8s — scale API independently |
|
||||
| **Frontend** | `Dockerfile.frontend` | nginx + static files + CM6 bundle | K8s — scale FE independently |
|
||||
|
||||
**Unified** bundles everything in one container (4-stage Docker build: vendor libs → CM6 bundle → Go backend → nginx runtime). Nginx serves static files and proxies `/api/*` and `/ws` to the Go backend running on `:8080`. Good for development and small deployments.
|
||||
|
||||
**Split (Backend + Frontend)** for production K8s: Ingress routes `/api/*` and `/ws` to the backend Service, everything else to the frontend Service. The frontend entrypoint (`docker-entrypoint-fe.sh`) handles `BASE_PATH` injection into `index.html` and dynamic nginx config generation at startup. Supports branding volume mounts at `/branding/`.
|
||||
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ Ingress / Traefik │
|
||||
│ ├─ /api/* → be-svc:8080 │
|
||||
│ ├─ /ws → be-svc:8080 │
|
||||
│ └─ /* → fe-svc:80 │
|
||||
└─────────────────────────┘
|
||||
│ │
|
||||
┌──────────▼──┐ ┌───────▼────────┐
|
||||
│ Backend │ │ Frontend │
|
||||
│ (Go :8080) │ │ (nginx :80) │
|
||||
│ replicas:N │ │ replicas:M │
|
||||
└──────┬──────┘ └────────────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ PostgreSQL │ (or SQLite for
|
||||
└─────────────┘ single-node)
|
||||
```
|
||||
Switchboard Core is a self-hosted extension platform. It provides identity,
|
||||
teams, permissions, storage, workflows, notifications, and a package system.
|
||||
Everything else — chat, AI providers, personas, knowledge bases, notes,
|
||||
tools — ships as installable extensions.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Persona-as-Trust-Boundary**: A persona (model + config + prompt) is the unit of access control. Users interact with personas, not raw provider configs. Admins control which models are visible; team admins control which personas their team can use.
|
||||
|
||||
2. **Roles vs Teams**: Clean separation between vertical permissions (Roles: admin, user) and horizontal visibility (Teams: organizational units). A user's role determines what they can *do*; their team membership determines what they can *see*.
|
||||
|
||||
3. **Store Layer**: All database access goes through typed Go interfaces (`store.Stores`). Handlers never write raw SQL. The store layer has two implementations — `store/postgres/` and `store/sqlite/` — selected at startup via `DB_DRIVER`. This enables Postgres for production and SQLite for single-node or air-gapped deployments.
|
||||
|
||||
4. **Scope Model**: Provider configs, personas, and model settings all use a three-value `scope` column: `global` (admin-managed, visible to all), `team` (team-admin-managed, visible to team), `personal` (user-managed, visible to owner). The `owner_id` column points to the owning user or team depending on scope.
|
||||
|
||||
5. **Capabilities Resolution**: Model capabilities (vision, tool calling, thinking, context window) are resolved through a three-tier priority chain: catalog DB (provider API sync, per-provider authoritative) → heuristic inference (regex patterns on model ID) → admin overrides (highest priority, from `capability_overrides` table). No static model table — the same model can have different capabilities on different providers. The catalog is populated by auto-fetch on provider creation and manual refresh. Admins can correct any field via the override endpoints.
|
||||
|
||||
6. **Channels as Execution Context**: Channels are the universal container for conversation state — messages, tool activity, notes, and artifacts all hang off a channel. Today channels are single-user direct chats. The architecture anticipates multi-participant channels (team members, anonymous visitors, AI personas) for workflow execution, without requiring changes to the message tree, tool framework, or streaming infrastructure.
|
||||
|
||||
## Workflow Architecture (v0.26.0 — Shipped)
|
||||
|
||||
The platform's existing primitives (teams, personas, channels, tools, notes)
|
||||
compose into a workflow engine where the channel is the execution context
|
||||
that moves through defined stages.
|
||||
|
||||
### Data Model
|
||||
|
||||
```
|
||||
Team
|
||||
└─ Workflow (admin-defined, team-scoped or global)
|
||||
├─ name, slug, description, branding (JSONB)
|
||||
├─ entry_mode (authenticated | public_link)
|
||||
├─ is_active, version (auto-incremented on edit)
|
||||
└─ Stages[] (ordered)
|
||||
├─ persona_id (which AI drives this stage)
|
||||
├─ system_prompt (stage-specific override)
|
||||
├─ form_template (JSONB — fields to collect)
|
||||
├─ history_mode (full | summary | fresh)
|
||||
└─ assignment_team_id (who can be assigned)
|
||||
|
||||
Channel (workflow instance)
|
||||
├─ workflow_id + workflow_version + current_stage
|
||||
├─ stage_data (JSONB — accumulated form data across stages)
|
||||
├─ workflow_status (active | completed | stale | cancelled)
|
||||
├─ participants[]
|
||||
│ ├─ anonymous visitor (session token)
|
||||
│ ├─ AI persona (per-stage, from workflow definition)
|
||||
│ └─ assigned team member (claimed from assignment queue)
|
||||
├─ messages (existing tree structure)
|
||||
└─ notes (stage data persisted as channel-scoped notes)
|
||||
|
||||
WorkflowAssignment (queue entry)
|
||||
├─ channel_id + stage + team_id
|
||||
├─ assigned_to (nullable — claimed by team member)
|
||||
└─ status (unassigned | claimed | completed)
|
||||
```
|
||||
|
||||
### How Existing Primitives Map
|
||||
|
||||
| Existing Primitive | Workflow Role |
|
||||
|-------------------|---------------|
|
||||
| **Persona** (system prompt + model) | Drives a workflow stage — the AI knows what to collect, when to route |
|
||||
| **Team** (members + roles) | Owns the workflow definition; members are assignable to channels |
|
||||
| **Channel** (messages + tree) | Execution instance of a workflow; full conversation history |
|
||||
| **Notes + Tools** | Structured data collection; `workflow_advance` tool triggers transitions, notes persist form data |
|
||||
| **EventBus + WebSocket** | Real-time notifications for assignment, stage transitions, new messages |
|
||||
| **Anonymous Sessions** (v0.24.3) | Visitors participate without user accounts |
|
||||
| **DenyVisitor** predicate | Scopes tool availability per participant type |
|
||||
|
||||
### Stage Transition Flow
|
||||
|
||||
```
|
||||
Visitor starts → Channel created (stage 0, persona bound)
|
||||
│
|
||||
▼
|
||||
Persona collects form data via conversation
|
||||
│
|
||||
▼
|
||||
Persona calls workflow_advance(data) → Stage notes created
|
||||
│ │
|
||||
▼ ▼
|
||||
Next stage: new persona bound If assignment_team_id set:
|
||||
(or workflow completes) → workflow_assignment created
|
||||
→ team member claims
|
||||
→ member + persona collaborate
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
- **Form template injection.** When `channelType == "workflow"`, the completion
|
||||
handler loads the current stage's `form_template` JSONB and injects a system
|
||||
prompt telling the persona what fields to collect and when to call
|
||||
`workflow_advance`.
|
||||
|
||||
- **Dialect-safe stage data merge.** `MergeWorkflowStageData()` reads existing
|
||||
JSON from the `stage_data` column, merges in Go, writes back. No
|
||||
Postgres-specific JSONB operators — works on both Postgres and SQLite.
|
||||
|
||||
- **Optimistic claim lock.** `UPDATE workflow_assignments SET assigned_to = $1
|
||||
WHERE id = $2 AND status = 'unassigned'` — if rows_affected == 0, another
|
||||
team member already claimed it.
|
||||
|
||||
- **Staleness sweep.** Background goroutine (1h tick) marks workflow instances
|
||||
as `stale` when `last_activity_at` exceeds `WORKFLOW_STALE_HOURS`.
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
server/
|
||||
├── main.go # Wiring: stores → handlers → routes
|
||||
├── config/config.go # Env-based configuration
|
||||
├── database/
|
||||
│ ├── database.go # Connection management (PG + SQLite)
|
||||
│ ├── migrate.go # Auto-migration on startup
|
||||
│ └── migrations/
|
||||
│ ├── 001_v016_schema.sql # Consolidated schema (PG)
|
||||
│ └── 002_v017_persona_kb.sql
|
||||
├── store/
|
||||
│ ├── interfaces.go # Store interfaces + shared types
|
||||
│ ├── postgres/ # Postgres implementations (22 stores)
|
||||
│ │ ├── stores.go # NewStores() constructor
|
||||
│ │ ├── provider.go # ProviderStore
|
||||
│ │ ├── catalog.go # CatalogStore
|
||||
│ │ ├── persona.go # PersonaStore
|
||||
│ │ ├── channel.go # ChannelStore
|
||||
│ │ ├── message.go # MessageStore
|
||||
│ │ ├── user.go # UserStore
|
||||
│ │ ├── team.go # TeamStore
|
||||
│ │ ├── note.go # NoteStore
|
||||
│ │ ├── knowledge.go # KnowledgeStore
|
||||
│ │ ├── memory.go # MemoryStore (CRUD + recall)
|
||||
│ │ ├── memory_hybrid.go # RecallHybrid (pgvector cosine)
|
||||
│ │ └── ...
|
||||
│ └── sqlite/ # SQLite implementations (22 stores)
|
||||
│ ├── stores.go # NewStores() constructor
|
||||
│ └── ... # Mirror of postgres/ with dialect adaptations
|
||||
├── models/models.go # Shared domain types
|
||||
├── capabilities/
|
||||
│ ├── intrinsic.go # Heuristic detection + three-tier resolution (catalog → heuristic → admin override)
|
||||
│ └── resolver.go # ModelsForUser() unified resolver
|
||||
├── compaction/ # Conversation summarization engine
|
||||
├── crypto/ # AES-256-GCM API key encryption
|
||||
├── events/ # EventBus + WebSocket hub
|
||||
├── extraction/ # Document text extraction pipeline
|
||||
├── health/ # Provider health tracking (v0.22.0), tool health + auto-disable (v0.22.4)
|
||||
├── routing/ # Policy-based request routing (v0.22.2)
|
||||
│ ├── types.go # Policy, Candidate, Context, Decision types
|
||||
│ ├── evaluator.go # Policy evaluation engine (4 policy types)
|
||||
│ ├── evaluator_test.go # 12 tests
|
||||
│ ├── fallback.go # RunWithFallback candidate dispatcher
|
||||
│ └── convert.go # DB model → routing type conversion
|
||||
│ ├── accumulator.go # In-memory counters, 60s flush, rate limits, tool health, auto-disable
|
||||
│ └── accumulator_test.go # Concurrent recording, flush isolation, threshold tests
|
||||
├── memory/ # Long-term memory extraction + scanning
|
||||
│ ├── extractor.go # Conversation analysis → fact extraction
|
||||
│ └── scanner.go # Background job: find channels needing extraction
|
||||
├── mentions/ # @mention parsing for multi-model routing
|
||||
│ └── parser.go # Parse() extracts + resolves @mentions against roster
|
||||
├── notifications/ # Notification service + email transport
|
||||
│ ├── service.go # Notify/NotifyMany, preference resolution, dispatch
|
||||
│ ├── email.go # SMTP transport (TLS/STARTTLS, multipart MIME)
|
||||
│ └── templates.go # HTML + plaintext email templates
|
||||
├── handlers/ # HTTP handlers (Gin)
|
||||
│ ├── auth.go # Login, register, refresh, logout
|
||||
│ ├── admin.go # User/config/model management
|
||||
│ ├── channels.go # Channel CRUD
|
||||
│ ├── messages.go # Message CRUD + forking
|
||||
│ ├── completion.go # Chat completions (SSE streaming)
|
||||
│ ├── stream_loop.go # Shared streaming + tool execution loop
|
||||
│ ├── capabilities.go # ModelHandler: model list + health enrichment + ResolveModelCaps (v0.22.3)
|
||||
│ ├── health_admin.go # Provider health + capability override admin endpoints
|
||||
│ ├── export.go # Markdown → PDF/DOCX conversion via pandoc (v0.22.4)
|
||||
│ ├── routing_admin.go # Routing policy CRUD + dry-run test (v0.22.2)
|
||||
│ ├── presets.go # Persona CRUD (all scopes)
|
||||
│ ├── apiconfigs.go # User provider config CRUD (BYOK)
|
||||
│ ├── teams.go # Team management
|
||||
│ ├── notes.go # Notes CRUD + search + graph + backlinks
|
||||
│ ├── knowledge.go # Knowledge base management
|
||||
│ ├── memory.go # Memory CRUD + review (user + admin)
|
||||
│ ├── memory_inject.go # BuildMemoryHint() for completion injection
|
||||
│ ├── notifications.go # Notification CRUD + preference endpoints
|
||||
│ ├── admin_email.go # Admin SMTP test email endpoint
|
||||
│ ├── channel_models.go # Channel model roster CRUD (multi-model)
|
||||
│ ├── model_prefs.go # User model visibility preferences
|
||||
│ └── ...
|
||||
├── notelinks/ # Wikilink extraction (regex → NoteLink structs)
|
||||
├── knowledge/ # KB chunking, embedding, search
|
||||
├── providers/ # LLM provider adapters
|
||||
│ ├── provider.go # Provider interface + ExtraBody/mergeExtraBody
|
||||
│ ├── registry.go # Data-driven type registry with metadata (v0.22.1)
|
||||
│ ├── profile.go # Profile schemas per provider type (v0.22.1)
|
||||
│ ├── hooks.go # PreRequest/PostStreamEvent transforms (v0.22.1)
|
||||
│ ├── hooks_test.go # 17 tests: all hooks, schemas, merge, nil safety
|
||||
│ ├── anthropic.go # Anthropic Messages API (+ extended thinking)
|
||||
│ ├── openai.go # OpenAI-compatible (+ ExtraBody merge)
|
||||
│ ├── openrouter.go
|
||||
│ └── venice.go
|
||||
├── middleware/ # Auth, admin, CORS, rate limiting
|
||||
├── storage/ # Blob storage (PVC + S3)
|
||||
└── tools/ # Built-in tool definitions
|
||||
```
|
||||
|
||||
## Store Layer Pattern
|
||||
|
||||
Every store follows the same pattern:
|
||||
|
||||
```go
|
||||
// Interface in store/interfaces.go
|
||||
type FooStore interface {
|
||||
Create(ctx context.Context, f *models.Foo) error
|
||||
GetByID(ctx context.Context, id string) (*models.Foo, error)
|
||||
Update(ctx context.Context, id string, patch models.FooPatch) error
|
||||
Delete(ctx context.Context, id string) error
|
||||
List(ctx context.Context, opts ListOptions) ([]models.Foo, int, error)
|
||||
}
|
||||
|
||||
// Implementation in store/postgres/foo.go
|
||||
type fooStore struct{ db *sql.DB }
|
||||
func (s *fooStore) Create(ctx context.Context, f *models.Foo) error { ... }
|
||||
```
|
||||
|
||||
Handlers receive `store.Stores` (a bundle of all store interfaces):
|
||||
|
||||
```go
|
||||
type AdminHandler struct { stores store.Stores }
|
||||
|
||||
func (h *AdminHandler) CreateUser(c *gin.Context) {
|
||||
// ...
|
||||
err := h.stores.Users.Create(c.Request.Context(), user)
|
||||
}
|
||||
```
|
||||
|
||||
## Capabilities Resolution Chain
|
||||
|
||||
When the system needs to know what a model can do (vision? tools? thinking?):
|
||||
|
||||
```
|
||||
1. model_catalog DB (exact match: model_id + provider_config_id)
|
||||
↓ miss
|
||||
2. model_catalog DB (any provider: same model_id)
|
||||
↓ miss
|
||||
3. Heuristic inference (name-based: "gpt-4-vision" → vision=true)
|
||||
```
|
||||
|
||||
The `capabilities.ModelsForUser()` function combines catalog entries, team personas, and user preferences into a single unified model list for the frontend.
|
||||
|
||||
## Scope / Ownership Model
|
||||
|
||||
```
|
||||
scope='global' → owner_id=NULL → Admin-managed, visible to all
|
||||
scope='team' → owner_id=team.id → Team-admin-managed, visible to team
|
||||
scope='personal' → owner_id=user.id → User-managed, visible to owner
|
||||
```
|
||||
|
||||
Used by: `provider_configs`, `personas`, `model_catalog` (visibility column adds `enabled`/`disabled`/`team` states on top), `projects`.
|
||||
|
||||
## Personas
|
||||
|
||||
Personas are named AI configurations: a system prompt, base model, provider
|
||||
config, and behavioral parameters (temperature, max tokens, thinking budget).
|
||||
Each persona has a unique `handle` field (e.g. `veronica-sharpe`) used for
|
||||
@mention routing.
|
||||
|
||||
**Handle Generation:** Auto-generated from name on create (`HandleFromName()`):
|
||||
lowercase, spaces→hyphens, alphanumeric only, max 50 chars. Editable via the
|
||||
persona form. Unique index prevents collisions.
|
||||
|
||||
**Scope Model:** Same as other resources — global, team, personal. Personal
|
||||
personas are only visible to the creator. Team and global personas are visible
|
||||
to all users with the appropriate access.
|
||||
|
||||
**Resolution Chain (completion):**
|
||||
```
|
||||
Explicit persona (req.PersonaID) → Project persona → @mention persona → dropdown selection
|
||||
```
|
||||
|
||||
**Memory:** Personas have `memory_enabled` and `memory_extraction_prompt` fields.
|
||||
When memory is enabled, the persona's memory scope is independent — facts
|
||||
extracted in conversations with Persona A are not visible to Persona B.
|
||||
|
||||
## Projects (v0.19.0)
|
||||
|
||||
Projects are organizational containers that group channels, knowledge bases,
|
||||
and notes. They follow the same scope model as other resources.
|
||||
|
||||
**KB Resolution Chain:**
|
||||
```
|
||||
Persona KBs → Project KBs → Channel KBs → Personal KBs
|
||||
```
|
||||
|
||||
When a channel belongs to a project, all KBs bound to that project are
|
||||
automatically included in both the system prompt hint (`BuildKBHint`) and
|
||||
tool-time search (`kbsearch`). This means adding a KB to a project makes it
|
||||
available to every channel in that project without per-channel configuration.
|
||||
|
||||
**Channel Association:** Channels have an optional `project_id` FK.
|
||||
The `project_channels` junction table maintains ordered membership with a
|
||||
UNIQUE constraint on `channel_id` (a channel can only belong to one project).
|
||||
`AddChannel` performs an atomic move: single transaction deletes from the
|
||||
old project, inserts into the new one, and updates the denormalized FK.
|
||||
|
||||
**Store Pattern:** `ProjectStore` interface with Postgres and SQLite
|
||||
implementations following the same conventions as other stores (see
|
||||
Store Layer Pattern above). Access checks via `UserCanAccess` enforce
|
||||
owner-or-team-member visibility.
|
||||
|
||||
## Schema Migration
|
||||
|
||||
Migrations are handled by the backend on startup (no separate migration job):
|
||||
|
||||
1. Creates `schema_migrations` table if absent
|
||||
2. Checks which migration files have been applied
|
||||
3. Applies any new `.sql` files in order
|
||||
|
||||
Postgres uses consolidated schemas (`001_v016_schema.sql` + incremental). SQLite uses application-generated UUIDs and `datetime()` instead of `gen_random_uuid()` and `now()`. Both drivers share the same migration tracking table.
|
||||
|
||||
## Notifications (v0.20.0)
|
||||
|
||||
Centralized notification infrastructure with real-time WebSocket delivery
|
||||
and optional email transport.
|
||||
|
||||
**Service Pattern:** All notification sources go through `notifications.Service`,
|
||||
never insert directly. `Notify()` persists to store, checks user preferences,
|
||||
pushes via WebSocket (if online), and dispatches email (if configured and
|
||||
opted-in). `NotifyMany()` fans out to multiple users (best-effort).
|
||||
|
||||
**Preference Resolution:**
|
||||
```
|
||||
Specific type pref → User wildcard '*' pref → System default (in_app=true, email=false)
|
||||
```
|
||||
|
||||
**Email Transport:** Optional SMTP via `EmailTransport`. Supports implicit TLS
|
||||
(port 465) and STARTTLS (port 587). Multipart MIME (HTML + plaintext).
|
||||
Templates rendered via Go `html/template` with instance branding. Async
|
||||
delivery via goroutine (failures logged, never block notification creation).
|
||||
Admin configures SMTP in platform settings; users toggle per-type delivery
|
||||
in Settings → Notifications.
|
||||
|
||||
**Cleanup:** Background goroutine deletes notifications older than retention
|
||||
period (default 90 days, configurable via admin settings).
|
||||
|
||||
## @mention Routing (v0.23.0)
|
||||
|
||||
Type `@handle` or `@model-id` in any chat to route the completion to a
|
||||
different persona or model. No roster, group, or channel setup required.
|
||||
|
||||
**Resolution (`resolveMention`):** Extracts the first `@token` from message
|
||||
content and queries the DB directly:
|
||||
|
||||
1. Persona handle — exact match against `personas.handle` (case-insensitive)
|
||||
2. Persona handle — prefix match (unambiguous only)
|
||||
3. Model ID — exact match against `model_catalog.model_id` (enabled + active provider)
|
||||
4. Model ID — prefix match (unambiguous only)
|
||||
|
||||
Persona resolution returns the persona's model, provider config, and system
|
||||
prompt. Model resolution returns the raw model ID and config — no persona
|
||||
character, no system prompt override.
|
||||
|
||||
**Context Boundaries:** When @mentioning a persona, a system message is
|
||||
injected before the user's message telling the target to ignore previous
|
||||
personas' styles. When no @mention is used but persona responses exist in
|
||||
history, a boundary tells the default model to respond in its own style.
|
||||
|
||||
**Participant Hint:** `buildParticipantHint()` injects a system message
|
||||
listing all @mentionable personas (handle + name + description). This tells
|
||||
the LLM who it can invoke, enabling AI-to-AI chaining.
|
||||
|
||||
**AI-to-AI Chaining:** After each completion, `chainIfMentioned()` runs
|
||||
`resolveMention()` on the assistant's response. If it @mentions another
|
||||
persona, a follow-up completion fires asynchronously. Result delivered via
|
||||
WebSocket (`message.created`). Self-mention blocked. Depth capped at 5.
|
||||
Same resolution function for user→LLM and LLM→LLM routing.
|
||||
|
||||
**Autocomplete (Frontend):** `ChannelModels.onInput()` triggers on `@` in
|
||||
the chat input, matching against all `App.models` (enabled, non-hidden) by
|
||||
persona handle, model ID, and display name. Popup shows avatar, name,
|
||||
`@handle` hint, and provider. Selection inserts `@handle` into the input.
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
Vanilla JavaScript, no framework. The frontend ships as static files served by nginx. The only build step is the CM6 editor bundle, compiled at Docker build time via esbuild (IIFE output, no runtime bundler).
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── index.html # Redirect stub → server-rendered surfaces
|
||||
├── sw.js # Service worker (offline cache, BUILD_HASH)
|
||||
├── manifest.json # PWA manifest
|
||||
├── css/
|
||||
│ ├── variables.css # Theme vars, reset, base styles
|
||||
│ ├── layout.css # App shell, workspace, sidebar, responsive
|
||||
│ ├── primitives.css # Buttons, forms, toasts, toggles, badges, icon-btn
|
||||
│ ├── modals.css # Modal system, debug, command palette, confirm
|
||||
│ ├── chat.css # Chat area, markdown, input, files, lightbox
|
||||
│ ├── panels.css # Side panel, preview, notes, graph
|
||||
│ ├── surfaces.css # Admin, settings, editor, projects, notes layouts
|
||||
│ ├── splash.css # Auth splash, PWA banner
|
||||
│ ├── chat-pane.css # ChatPane component styles
|
||||
│ ├── pane-container.css # PaneContainer resize/tab styles
|
||||
│ ├── user-menu.css # UserMenu flyout styles
|
||||
│ ├── editor-surface.css # Editor surface 3-pane layout
|
||||
│ ├── channel-models.css # @mention autocomplete, model roster
|
||||
│ ├── memory.css # Memory settings + admin review
|
||||
│ ├── notifications.css # Notification dropdown + list
|
||||
│ ├── notification-prefs.css # SMTP config, notification pref table
|
||||
│ ├── persona-kb.css # Persona KB picker panel
|
||||
│ ├── admin-surfaces.css # Admin surface management
|
||||
│ └── tool-grants.css # Persona tool grants checkbox list
|
||||
├── js/
|
||||
│ ├── ── Primitives (base.html, all surfaces) ──
|
||||
│ ├── app-state.js # Global state (App, API refs)
|
||||
│ ├── api.js # HTTP client with token refresh
|
||||
│ ├── events.js # Labeled event bus + WebSocket bridge
|
||||
│ ├── ui-primitives.js # esc(), componentMixin, Providers, Roles,
|
||||
│ │ # renderCapBadges, renderProviderForm/List,
|
||||
│ │ # renderRoleConfig, renderUsageDashboard,
|
||||
│ │ # showConfirm, showPrompt, openModal/closeModal,
|
||||
│ │ # Theme, mountAvatarUpload
|
||||
│ ├── ui-format.js # Markdown rendering, code blocks, time formatting
|
||||
│ ├── ui-core.js # DOM rendering, sidebar, navigation, toast
|
||||
│ ├── pages.js # Page routing, splash, login
|
||||
│ │
|
||||
│ ├── ── Components (base.html, all surfaces) ──
|
||||
│ ├── user-menu.js # UserMenu template+factory component
|
||||
│ ├── model-selector.js # ModelSelector template+factory component
|
||||
│ ├── file-tree.js # FileTree template+factory component
|
||||
│ ├── code-editor.js # CodeEditor template+factory component (CM6)
|
||||
│ ├── note-editor.js # NoteEditor template+factory component
|
||||
│ ├── chat-pane.js # ChatPane template+factory component
|
||||
│ ├── pane-container.js # PaneContainer (resizable multi-pane layout)
|
||||
│ ├── debug.js # Debug panel + diagnostics
|
||||
│ ├── repl.js # Browser REPL console
|
||||
│ │
|
||||
│ ├── ── Surface: Chat ──
|
||||
│ ├── app.js # Chat surface boot, auth, WebSocket
|
||||
│ ├── chat.js # Chat message send/receive, stream
|
||||
│ ├── extensions.js # Extension framework + sandbox
|
||||
│ ├── panels.js # Side panel registry
|
||||
│ ├── ui-settings.js # Settings modal tabs (chat surface)
|
||||
│ ├── ui-admin.js # Admin navigation + section routing
|
||||
│ ├── admin-handlers.js # Admin CRUD + extension editors
|
||||
│ ├── admin-scaffold.js # Admin section HTML scaffolding
|
||||
│ ├── settings-handlers.js # User settings CRUD, command palette
|
||||
│ ├── tokens.js # Input token counting + context budget
|
||||
│ ├── files.js # File upload, paste-to-file, lightbox
|
||||
│ ├── notes.js # Notes panel CRUD + graph + daily notes
|
||||
│ ├── note-graph.js # Canvas force-directed graph visualization
|
||||
│ ├── projects-ui.js # Project sidebar, context menu, DnD
|
||||
│ ├── channel-models.js # Channel model roster + @mention AC
|
||||
│ ├── tools-toggle.js # Tool enable/disable popup
|
||||
│ ├── knowledge-ui.js # Knowledge base picker + management
|
||||
│ ├── persona-kb.js # Persona KB assignment panel
|
||||
│ ├── memory-ui.js # Memory settings + admin review
|
||||
│ ├── notifications.js # Notification bell + dropdown
|
||||
│ ├── notification-prefs.js # SMTP config + notification prefs
|
||||
│ │
|
||||
│ ├── ── Surface: Editor ──
|
||||
│ ├── editor-surface.js # Editor 3-pane boot, workspace management
|
||||
│ │
|
||||
│ ├── ── Surface: Admin ──
|
||||
│ ├── admin-surfaces.js # Surface management (admin only)
|
||||
│ │
|
||||
│ ├── ── Surface: Login ──
|
||||
│ ├── pages-splash.js # Login/register splash page
|
||||
│ │
|
||||
│ └── __tests__/ # Node.js test suite (node --test)
|
||||
├── editor/ # CM6 source (compiled at build time)
|
||||
│ ├── package.json # CM6 + language mode dependencies
|
||||
│ ├── build.mjs # esbuild script → IIFE bundle
|
||||
│ ├── index.mjs # Bundle entrypoint (window.CM)
|
||||
│ ├── chat-input.mjs # Chat input factory (markdown, code blocks)
|
||||
│ ├── code-editor.mjs # Code editor factory (syntax, keybindings)
|
||||
│ ├── note-editor.mjs # Note editor factory (markdown + wikilinks)
|
||||
│ ├── wikilink.mjs # CM6 wikilink extension
|
||||
│ └── theme.mjs # CM6 themes (dark/light)
|
||||
└── vendor/ # Vendored libraries (loaded in base.html)
|
||||
├── marked.min.js # Markdown renderer
|
||||
├── purify.min.js # DOMPurify (XSS protection)
|
||||
└── codemirror/
|
||||
└── codemirror.bundle.js # CM6 bundle (~295KB min, ~90KB gzip)
|
||||
```
|
||||
|
||||
### CM6 Integration
|
||||
|
||||
The CodeMirror 6 bundle provides three factory functions exposed on `window.CM`:
|
||||
|
||||
**`CM.chatInput(target, opts)`** — Markdown-mode editor for the chat input area. Features: auto-growing height, Enter=send / Shift+Enter=newline, WYSIWYG fenced code block decorations (visual container with monospace font and accent border), inline code shortcut (Ctrl/Cmd+E), spell check. Always uses standard keybindings.
|
||||
|
||||
**`CM.codeEditor(target, opts)`** — Full-featured code editor for the admin extension panel (JSON manifests, JavaScript scripts). Features: line numbers, bracket matching, search/replace, fold gutter, 10 bundled language modes. Supports Vim and Emacs keybindings via user preference (reconfigurable at runtime via compartments).
|
||||
|
||||
**`CM.noteEditor(target, opts)`** — Rich markdown editor for the notes panel. Features: heading size rendering (h1–h3), blockquote styling, fenced code block decorations, `[[wikilink]]` autocomplete (triggered by `[[`), wikilink chip rendering (clickable, styled by link/transclusion type), search/replace. `onLink` callback for navigating to linked notes; `linkCompleter` async callback for autocomplete results.
|
||||
|
||||
Both factories return a clean API: `getValue()`, `setValue()`, `focus()`, `destroy()`. The `ChatInput` abstraction in `chat.js` wraps the CM6 instance with a textarea fallback — all callsites use `ChatInput.getValue()` etc., never raw DOM access.
|
||||
|
||||
**Graceful degradation**: Every integration point checks `window.CM` before using CM6. If the bundle fails to load (air-gapped without the build step, CDN issues), the app falls back to native `<textarea>` with zero breakage.
|
||||
|
||||
### Theme System
|
||||
|
||||
CSS variables define the color palette in `:root` (dark, default) and `[data-theme="light"]` (light override). All UI components — including CM6 editors — reference these variables, so theme switches propagate instantly without editor reconfiguration.
|
||||
|
||||
The appearance settings offer three modes: Light, Dark, System. System mode listens to `prefers-color-scheme` and re-evaluates on OS theme change. Theme changes emit `theme.changed` on the EventBus; CM6 code editors toggle the `oneDark` syntax theme via compartment reconfiguration.
|
||||
|
||||
### Communication Pattern
|
||||
|
||||
`app.js` calls `API.*` methods, updates state, then calls `UI.*` methods to render. `events.js` handles real-time updates via WebSocket. No framework, no virtual DOM, no reactive bindings.
|
||||
|
||||
## Security Model
|
||||
|
||||
- **Auth**: JWT access tokens (short-lived) + refresh tokens (DB-stored, revocable)
|
||||
- **Admin Bootstrap**: `SWITCHBOARD_ADMIN_USERNAME`/`PASSWORD` env vars create/update admin on every startup (K8s secret pattern)
|
||||
- **API Key Storage**: Provider API keys encrypted with AES-256-GCM. Two-tier: org keys use `ENCRYPTION_KEY` env var, personal BYOK keys use per-user encryption keys (admins cannot recover)
|
||||
- **Policies**: Boolean flags in `global_settings` table control registration, BYOK, team providers, etc.
|
||||
- **Audit**: All admin operations logged to `audit_log` with actor, action, resource type/ID, and diff
|
||||
- **Banner**: Environment classification banner configurable via admin settings (text, color, position)
|
||||
- **CORS**: `CORS_ALLOWED_ORIGINS` env var controls which origins can make
|
||||
cross-origin requests (HTTP and WebSocket). Comma-separated list of
|
||||
allowed origins, e.g. `https://switchboard.corp,https://admin.corp`.
|
||||
Behavior by environment:
|
||||
- **Production** (`ENVIRONMENT=production`): if unset, defaults to
|
||||
same-origin only (no `Access-Control-Allow-Origin` header). Set
|
||||
explicitly to the domain(s) serving the frontend.
|
||||
- **Development/test**: if unset, defaults to `*` (all origins).
|
||||
- **Explicit `*`**: allowed but logs a startup warning. Not recommended
|
||||
for production — restricts nothing and exposes the API to any origin.
|
||||
The WebSocket upgrader's `CheckOrigin` respects the same setting.
|
||||
Connections from origins not in the list are rejected at upgrade time.
|
||||
- **No sensitive terminology**: Banner system avoids classification-related terms for security compliance
|
||||
|
||||
## File Storage
|
||||
|
||||
Blob storage for attachments, extracted text, and (future) knowledge base documents.
|
||||
|
||||
**Interface**: `storage.ObjectStore` — `Put`, `Get`, `Delete`, `DeletePrefix`, `Exists`, `Healthy`, `Stats`, `Backend`. All handlers use the interface; backend is selected at startup.
|
||||
|
||||
**Backends**:
|
||||
|
||||
| Backend | Config | Use Case |
|
||||
|---------|--------|----------|
|
||||
| **PVC** | `STORAGE_BACKEND=pvc` + `STORAGE_PATH` | Single-node, dev, docker-compose. Local filesystem with atomic writes (temp+rename). |
|
||||
| **S3** | `STORAGE_BACKEND=s3` + `S3_ENDPOINT`, `S3_BUCKET`, credentials | Multi-node production. MinIO, Ceph RGW, AWS S3. Uses minio-go v7. |
|
||||
| **Auto** | `STORAGE_BACKEND=` (empty) | Tries PVC at `STORAGE_PATH`; disables if not writable. |
|
||||
|
||||
**Key layout**: `attachments/{channel_id}/{attachment_id}_{filename}`. S3 prefix (`S3_PREFIX`) prepended for shared buckets.
|
||||
|
||||
**PVC always mounted**: Even with S3 backend, the PVC mount at `STORAGE_PATH` is needed for the extraction queue's local scratch directory (`processing/{id}/status.json`). With S3, the PVC can be small (1Gi) — only transient coordination state, not blobs.
|
||||
|
||||
**Admin panel**: Settings → Storage tab shows backend type, health status, file count, total size, endpoint/bucket (S3) or path (PVC), orphan detection and cleanup.
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
v0.9 maintains backward-compatible API routes:
|
||||
|
||||
| Old Route | New Handler | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| `/api/v1/presets` | PersonaHandler | Returns both `personas` and `presets` keys |
|
||||
| `/api/v1/api-configs` | ProviderConfigHandler | Unchanged route, new implementation |
|
||||
| `/api/v1/models` | ModelHandler.ListEnabledModels | Alias for `/models/enabled` |
|
||||
**KISS.** Every kernel feature earns its place by being required by two or more
|
||||
extensions or by being impossible to implement outside the kernel. When in
|
||||
doubt, leave it out and let an extension handle it.
|
||||
|
||||
JSON field rename: `api_config_id` → `provider_config_id` in channel and completion request/response bodies. Frontend updated to match.
|
||||
**Store interface discipline.** All database access flows through the store
|
||||
interface layer. No raw SQL in handlers, middleware, or sandbox code.
|
||||
PostgreSQL and SQLite are first-class peers — every query compiles and
|
||||
passes tests on both.
|
||||
|
||||
**Two-track execution.** Go for platform operations (auth, migrations,
|
||||
package lifecycle). Starlark for custom admin/team logic (extension hooks,
|
||||
workflow automation, triggers). The boundary is permanent.
|
||||
|
||||
**Extensions all the way down.** The platform ships with a kernel and a
|
||||
package installer. Surfaces (UI pages), tools, filters, triggers, and
|
||||
providers are all packages. The editor, chat, and admin UI will themselves
|
||||
be installable surface packages.
|
||||
|
||||
## Kernel Components
|
||||
|
||||
### Identity & Auth
|
||||
|
||||
Users, refresh tokens, OIDC, mTLS, builtin password auth. The kernel owns
|
||||
the user lifecycle because extensions need a stable identity to key
|
||||
permissions, settings, and data against.
|
||||
|
||||
Auth modes: `builtin` (password), `mtls` (client cert), `oidc` (Keycloak
|
||||
et al.). Mode is set at deploy time via `AUTH_MODE` env var.
|
||||
|
||||
### Teams & Groups
|
||||
|
||||
Teams provide horizontal isolation — users see only their team's resources.
|
||||
Groups provide vertical permissions — what actions a user can perform.
|
||||
|
||||
The `Everyone` group (well-known UUID) grants baseline permissions to all
|
||||
authenticated users without an explicit membership row. Current kernel
|
||||
permissions: `extension.use`, `extension.install`, `workflow.create`,
|
||||
`workflow.submit`, `admin.view`, `token.unlimited`.
|
||||
|
||||
### Packages
|
||||
|
||||
The unified registry for all installable content. A package is a surface
|
||||
(routable UI page), an extension (Starlark hooks/tools/pipes), a library
|
||||
(shared dependency), or a workflow definition. Package types:
|
||||
|
||||
| Type | What it provides |
|
||||
|------------|-----------------------------------------------|
|
||||
| `surface` | A routable page rendered in the shell viewport |
|
||||
| `extension`| Starlark hooks, tools, API routes, DB tables |
|
||||
| `full` | Both surface and extension |
|
||||
| `workflow` | Bundled workflow definition |
|
||||
| `library` | Shared code imported by other packages |
|
||||
|
||||
Tiers: `browser` (JS only), `starlark` (sandboxed server-side),
|
||||
`sidecar` (container, future).
|
||||
|
||||
### Starlark Sandbox
|
||||
|
||||
Extensions declare capabilities in their manifest. The admin grants or
|
||||
denies each capability. At runtime the sandbox injects only the modules
|
||||
the extension has been granted:
|
||||
|
||||
- `db` — namespaced table CRUD (`ext_{pkg_id}_{table}`)
|
||||
- `http` — SSRF-safe outbound HTTP
|
||||
- `notifications` — push notifications to users
|
||||
- `secrets` — read connection credentials from the vault
|
||||
- `api` — register extension HTTP routes (`/s/:slug/api/*path`)
|
||||
|
||||
The sandbox cannot spawn goroutines, access the filesystem, or import
|
||||
arbitrary Go packages. It runs with a CPU budget and memory ceiling.
|
||||
|
||||
### Workflows
|
||||
|
||||
Staged processes with form collection, human review, and webhooks.
|
||||
Workflow definitions live in the kernel because they orchestrate teams,
|
||||
permissions, and notifications — all kernel concerns.
|
||||
|
||||
Stages support four modes: `form_only`, `form_chat`, `review`, `custom`.
|
||||
The `custom` mode delegates rendering to a surface package, proving the
|
||||
extension stack end-to-end.
|
||||
|
||||
Workflow *instances* are being redesigned. The old channel-based model
|
||||
(v0.38) is removed. New instance storage will use extension data tables
|
||||
or a dedicated kernel table — TBD in v0.2.0.
|
||||
|
||||
### Connections
|
||||
|
||||
Scoped credential storage for extensions. Global (admin-managed), team,
|
||||
or personal scope. Secrets are AES-256-GCM encrypted at rest using the
|
||||
platform encryption key or the user's vault key (BYOK).
|
||||
|
||||
Connection types are declared by packages. Multiple packages can share
|
||||
a connection type (e.g., both a GitHub extension and a CI extension
|
||||
declare type `github`).
|
||||
|
||||
### Triggers (planned — v0.2.0)
|
||||
|
||||
Three trigger types invoke extension Starlark handlers:
|
||||
|
||||
| Trigger | Source | Kernel primitive |
|
||||
|-----------|-------------------------|----------------------------|
|
||||
| `time` | Cron expression | Ticker goroutine + dispatch |
|
||||
| `webhook` | Inbound HTTP | Ext API route sugar |
|
||||
| `event` | Internal event bus | Subscription registry |
|
||||
|
||||
All three converge to `sandbox.Call(handler, triggerContext)`. The
|
||||
extension doesn't know or care how it was invoked.
|
||||
|
||||
Tasks (the old scheduler) are rebuilt as a Starlark extension on top
|
||||
of these three primitives. This validates the extension stack and
|
||||
removes ~3,400 lines from the kernel.
|
||||
|
||||
### Event Bus
|
||||
|
||||
Server-sent events to WebSocket clients. Kernel prefixes:
|
||||
`user.*`, `team.*`, `workflow.*`, `notification.*`, `presence.*`,
|
||||
`extension.*`, `admin.*`, `system.*`.
|
||||
|
||||
Extensions will subscribe to event patterns at install time (v0.2.0).
|
||||
Match expressions start as exact strings, grow to globs later.
|
||||
|
||||
### Storage & Notifications
|
||||
|
||||
**Object storage**: PVC (local disk) or S3-compatible. Used by extensions
|
||||
for file uploads, package assets, and blob storage.
|
||||
|
||||
**Notifications**: In-app notification bell with per-type preferences.
|
||||
Extension packages can push notifications via the sandbox module.
|
||||
|
||||
## Data Layer
|
||||
|
||||
Dual-store: PostgreSQL (production) and SQLite (dev/test/edge).
|
||||
27 kernel tables across 9 migration files per dialect:
|
||||
|
||||
| Migration | Tables |
|
||||
|-----------|--------|
|
||||
| 001_core | users, refresh_tokens, platform_policies, global_settings, user_presence, oidc_auth_state |
|
||||
| 002_teams | teams, team_members, groups, group_members |
|
||||
| 003_packages | packages, package_user_settings, extension_permissions, ext_data_tables |
|
||||
| 004_connections | ext_connections, ext_dependencies, resource_grants |
|
||||
| 005_notifications | notifications, notification_preferences |
|
||||
| 006_audit | audit_log |
|
||||
| 007_workflows | workflows, workflow_stages, workflow_versions |
|
||||
| 008_ha | ws_tickets, rate_limit_counters |
|
||||
|
||||
SQLite parity rules: `boolToInt` for boolean binding, `store.NewID()` for
|
||||
INSERT RETURNING, no `NULLS FIRST`, no boolean literals, no `$N` reuse,
|
||||
`database.ST()`/`database.SNT()` wrappers for time scanning.
|
||||
|
||||
## Frontend
|
||||
|
||||
Preact (3KB) + htm (tagged template literals). No build step, no bundler
|
||||
(except CM6 via esbuild). IIFE/global-namespace pattern with
|
||||
`sb.register()`/`sb.ns()`.
|
||||
|
||||
The shell loads surfaces into a viewport. Extensions use `window.html`
|
||||
and `window.preact` directly. Hooks via `window.hooks`. Vendor libs
|
||||
(marked.js, DOMPurify, KaTeX, CodeMirror 6) baked into the image.
|
||||
|
||||
## Deployment
|
||||
|
||||
Single Docker image: Go binary + migrations + frontend assets + vendor
|
||||
libs. Kubernetes deployment with 3-node PG cluster. CI via Gitea Actions
|
||||
with DaemonSet DinD runners testing both PG and SQLite pipelines.
|
||||
|
||||
Registry: `registry.gobha.me:5000/xcaliber/switchboard-core`
|
||||
Namespace: `gobha-ai-chat`
|
||||
|
||||
Reference in New Issue
Block a user