core/docs/ARCHITECTURE.md

# Switchboard Core — Architecture

Switchboard Core is a self-hosted extension platform. It provides identity,
teams, permissions, storage, workflows, notifications, and a package system.
Everything else — chat, AI providers, personas, knowledge bases, notes,
tools — ships as installable extensions.

## Design Principles

**KISS.** Every kernel feature earns its place by being required by two or more
extensions or by being impossible to implement outside the kernel. When in
doubt, leave it out and let an extension handle it.

**Store interface discipline.** All database access flows through the store
interface layer. No raw SQL in handlers, middleware, or sandbox code.
PostgreSQL and SQLite are first-class peers — every query compiles and
passes tests on both.

**Two-track execution.** Go for platform operations (auth, migrations,
package lifecycle). Starlark for custom admin/team logic (extension hooks,
workflow automation, triggers). The boundary is permanent.

**Extensions all the way down.** The platform ships with a kernel and a
package installer. Surfaces (UI pages), tools, filters, triggers, and
providers are all packages. The editor, chat, and admin UI will themselves
be installable surface packages.

## System Architecture Overview

```mermaid
graph TD
    Browser["Browser (Preact + htm)"]
    Server["Go Server (Gin)"]
    DB["Database (SQLite / Postgres)"]

    Browser -->|HTTP / WebSocket| Server

    subgraph Server Layers
        Handlers["HTTP Handlers"]
        Sandbox["Starlark Sandbox"]
        Store["Store Interface"]
        EventBus["Event Bus"]
    end

    Server --> Handlers
    Handlers --> Store
    Handlers --> Sandbox
    Sandbox --> Store
    Handlers --> EventBus
    Store --> DB
    EventBus -->|SSE / WS| Browser
```

## Kernel Components

### Identity & Auth

Users, refresh tokens, OIDC, mTLS, builtin password auth. The kernel owns
the user lifecycle because extensions need a stable identity to key
permissions, settings, and data against.

Auth modes: `builtin` (password), `mtls` (client cert), `oidc` (Keycloak
et al.). Mode is set at deploy time via `AUTH_MODE` env var.

### Request Flow

```mermaid
sequenceDiagram
    participant C as Client
    participant CORS as CORS Middleware
    participant Auth as Auth Middleware
    participant H as Handler
    participant S as Store
    participant DB as Database

    C->>CORS: HTTP Request
    CORS->>Auth: Pass through
    Auth->>Auth: Validate token / session
    alt Unauthorized
        Auth-->>C: 401 JSON error
    end
    Auth->>H: Authenticated context
    H->>S: Store method call
    S->>DB: SQL query
    DB-->>S: Rows
    S-->>H: Typed result
    H-->>C: JSON response
```

### Teams & Groups

Teams provide horizontal isolation — users see only their team's resources.
Groups provide vertical permissions — what actions a user can perform.

The `Everyone` group (well-known UUID) grants baseline permissions to all
authenticated users without an explicit membership row. Current kernel
permissions: `extension.use`, `extension.install`, `workflow.create`,
`workflow.submit`, `admin.view`, `token.unlimited`.

### Packages

The unified registry for all installable content. A package is a surface
(routable UI page), an extension (Starlark hooks/tools/pipes), a library
(shared dependency), or a workflow definition. Package types:

| Type       | What it provides                              |
|------------|-----------------------------------------------|
| `surface`  | A routable page rendered in the shell viewport |
| `extension`| Starlark hooks, tools, API routes, DB tables   |
| `full`     | Both surface and extension                     |
| `workflow`  | Bundled workflow definition                    |
| `library`  | Shared code imported by other packages         |

Tiers: `browser` (JS only), `starlark` (sandboxed server-side),
`sidecar` (container, future).

#### Extension Lifecycle

```mermaid
graph TD
    Upload["Package Upload (.pkg)"]
    Parse["Manifest Parse"]
    Insert["DB Insert (packages table)"]
    Enable["Admin Enable Toggle"]
    Mount["Surface Mount (/s/:slug)"]
    SDK["SDK Boot"]
    Render["Renderer Registration"]

    Upload --> Parse
    Parse --> Insert
    Insert --> Enable
    Enable --> Mount
    Mount --> SDK
    SDK --> Render
```

### Starlark Sandbox

Extensions declare capabilities in their manifest. The admin grants or
denies each capability. At runtime the sandbox injects only the modules
the extension has been granted:

- `db` — namespaced table CRUD (`ext_{pkg_id}_{table}`)
- `http` — SSRF-safe outbound HTTP
- `notifications` — push notifications to users
- `secrets` — read connection credentials from the vault
- `api` — register extension HTTP routes (`/s/:slug/api/*path`)

The sandbox cannot spawn goroutines, access the filesystem, or import
arbitrary Go packages. It runs with a CPU budget and memory ceiling.

### Workflows

Staged processes with form collection, human review, and webhooks.
Workflow definitions live in the kernel because they orchestrate teams,
permissions, and notifications — all kernel concerns.

Stages support four modes: `form_only`, `form_chat`, `review`, `custom`.
The `custom` mode delegates rendering to a surface package, proving the
extension stack end-to-end.

Workflow *instances* are being redesigned. The old channel-based model
(v0.38) is removed. New instance storage will use extension data tables
or a dedicated kernel table — TBD in v0.2.0.

### Connections

Scoped credential storage for extensions. Global (admin-managed), team,
or personal scope. Secrets are AES-256-GCM encrypted at rest using the
platform encryption key or the user's vault key (BYOK).

Connection types are declared by packages. Multiple packages can share
a connection type (e.g., both a GitHub extension and a CI extension
declare type `github`).

### Triggers (planned — v0.2.0)

Three trigger types invoke extension Starlark handlers:

| Trigger   | Source                  | Kernel primitive           |
|-----------|-------------------------|----------------------------|
| `time`    | Cron expression         | Ticker goroutine + dispatch |
| `webhook` | Inbound HTTP            | Ext API route sugar         |
| `event`   | Internal event bus      | Subscription registry       |

All three converge to `sandbox.Call(handler, triggerContext)`. The
extension doesn't know or care how it was invoked.

Tasks (the old scheduler) are rebuilt as a Starlark extension on top
of these three primitives. This validates the extension stack and
removes ~3,400 lines from the kernel.

### Event Bus

Server-sent events to WebSocket clients. Kernel prefixes:
`user.*`, `team.*`, `workflow.*`, `notification.*`, `presence.*`,
`extension.*`, `admin.*`, `system.*`.

Extensions will subscribe to event patterns at install time (v0.2.0).
Match expressions start as exact strings, grow to globs later.

#### Realtime Event Flow

```mermaid
sequenceDiagram
    participant C as Client
    participant WS as WebSocket Hub
    participant PG as PG LISTEN/NOTIFY
    participant Other as Other Node

    C->>WS: WS connect + subscribe
    Note over WS: Local fan-out to subscribers
    WS-->>C: Event push

    Other->>PG: NOTIFY channel, payload
    PG->>WS: LISTEN callback
    WS-->>C: Fan-out to local subscribers

    WS->>PG: NOTIFY channel, payload
    PG->>Other: LISTEN callback
```

### Storage & Notifications

**Object storage**: PVC (local disk) or S3-compatible. Used by extensions
for file uploads, package assets, and blob storage.

**Notifications**: In-app notification bell with per-type preferences.
Extension packages can push notifications via the sandbox module.

## Data Layer

Dual-store: PostgreSQL (production) and SQLite (dev/test/edge).
27 kernel tables across 9 migration files per dialect:

| Migration | Tables |
|-----------|--------|
| 001_core  | users, refresh_tokens, platform_policies, global_settings, user_presence, oidc_auth_state |
| 002_teams | teams, team_members, groups, group_members |
| 003_packages | packages, package_user_settings, extension_permissions, ext_data_tables |
| 004_connections | ext_connections, ext_dependencies, resource_grants |
| 005_notifications | notifications, notification_preferences |
| 006_audit | audit_log |
| 007_workflows | workflows, workflow_stages, workflow_versions |
| 008_ha | ws_tickets, rate_limit_counters |

SQLite parity rules: `boolToInt` for boolean binding, `store.NewID()` for
INSERT RETURNING, no `NULLS FIRST`, no boolean literals, no `$N` reuse,
`database.ST()`/`database.SNT()` wrappers for time scanning.

## Settings Cascade

```mermaid
graph TD
    Global["Global Scope (admin)"]
    Team["Team Override"]
    User["User Override"]
    RBAC{"RBAC Gate: who can set at what scope?"}
    Flag{"user_overridable flag"}
    Effective["Effective Value"]

    Global --> RBAC
    RBAC -->|allowed| Team
    RBAC -->|denied| Effective
    Team --> Flag
    Flag -->|true| User
    Flag -->|false| Effective
    User --> Effective
```

## Frontend

Preact (3KB) + htm (tagged template literals). No build step, no bundler
(except CM6 via esbuild). IIFE/global-namespace pattern with
`sb.register()`/`sb.ns()`.

The shell loads surfaces into a viewport. Extensions use `window.html`
and `window.preact` directly. Hooks via `window.hooks`. Vendor libs
(marked.js, DOMPurify, KaTeX, CodeMirror 6) baked into the image.

## Deployment

Single Docker image: Go binary + migrations + frontend assets + vendor
libs. Kubernetes deployment with 3-node PG cluster. CI via Gitea Actions
with DaemonSet DinD runners testing both PG and SQLite pipelines.

Registry: `registry.gobha.me:5000/xcaliber/switchboard-core`
Namespace: `gobha-ai-chat`

### Cluster Topology

```mermaid
graph TD
    N1["Node 1"]
    N2["Node 2"]
    N3["Node 3"]
    PG["PG UNLOGGED Table (cluster_nodes)"]
    Sweep["Stale Sweep Goroutine"]
    Notify["PG LISTEN/NOTIFY"]

    N1 -->|heartbeat INSERT/UPDATE| PG
    N2 -->|heartbeat INSERT/UPDATE| PG
    N3 -->|heartbeat INSERT/UPDATE| PG

    Sweep -->|DELETE nodes not seen| PG
    PG -->|join/leave events| Notify
    Notify --> N1
    Notify --> N2
    Notify --> N3
```