# Chat Switchboard — Extension System Specification

**Version:** 0.4
**Status:** All tiers implemented (Browser v0.11.0, Starlark v0.25.0, Sidecar v0.26.0). v0.30.2 adds workflow packages and SDK.
**See also:** [ARCHITECTURE.md](ARCHITECTURE.md), [ROADMAP.md](ROADMAP.md), [PACKAGES.md](PACKAGES.md), [SDK.md](SDK.md)

---

## 1. Philosophy

Chat Switchboard is a substrate, not an application. The core provides:
authentication, provider routing, message persistence, an event bus, and a
rendering surface. Everything else — editing, writing, cluster management,
cost tracking, custom renderers — is an extension.

The goal is that someone with a problem and some JS (or Go, or Python) can
solve it without forking the project. The "modes" — Editor, Article, Chat,
Cluster Manager — are extensions that register surfaces, tools, and event
handlers. This project doesn't have to build all of them. It has to make
them possible.

---

## 2. Extension Tiers

| | Tier 0: Browser | Tier 1: Starlark | Tier 2: Sidecar |
|---|---|---|---|
| **Runs in** | User's browser | Go server (embedded) | Separate container |
| **Language** | JavaScript | Starlark (Python subset) | Any |
| **Deployed by** | User or Admin push | Admin | Admin |
| **Latency** | Zero (client-side) | Low (in-process) | Network hop |
| **Can access** | DOM, EventBus, user context | Message data, DB reads (sandboxed) | Anything (HTTP, filesystem, network) |
| **Cannot access** | Server internals, other users' data | Network, filesystem, raw SQL | N/A (full access) |
| **Trust model** | Same-origin; user-scoped or admin-pushed | Starlark sandbox (no I/O) | Container isolation |
| **Use cases** | UI, rendering, shortcuts, client tools, modes | Routing rules, message transforms, logging | RAG, external APIs, webhooks, heavy compute |

All three tiers communicate through the EventBus. A browser extension
publishes `tool.result.{callId}` the same way a sidecar does — the bus
doesn't care where the event originated.

**Admin-pushed vs User-installed (Tier 0):**
Admin-pushed extensions load for all users (like a managed browser
extension policy). User-installed extensions load from user settings
and only affect that user's session. Both use the same runtime, same
manifest format. The difference is governance, not execution.

---

## 3. Manifest Format

Every extension, regardless of tier, is described by a manifest:

```json
{
  "id": "cost-tracker",
  "name": "Cost Tracker",
  "version": "1.0.0",
  "tier": "browser",
  "author": "jeff",
  "description": "Real-time token counting and cost estimation",

  "permissions": [
    "events:chat.message.*",
    "events:model.selected",
    "dom:input-area",
    "storage:local"
  ],

  "entry": "cost-tracker.js",

  "hooks": {
    "chat.message.send":    { "priority": 10, "async": false },
    "chat.message.received": { "priority": 50, "async": true }
  },

  "tools": [],
  "surfaces": [],

  "settings": {
    "showInline": {
      "type": "boolean",
      "label": "Show cost inline",
      "default": true
    }
  }
}
```

### 3.1 Fields

- **id**: Unique identifier. Namespaced by convention (`jeff.cost-tracker`).
- **tier**: `browser` | `starlark` | `sidecar`
- **permissions**: What the extension needs. The loader enforces these.
  Undeclared access is blocked (Tier 0 via proxy, Tier 1 via sandbox,
  Tier 2 via API scoping).
- **entry**: Browser: JS file path. Starlark: `.star` file. Sidecar:
  endpoint URL or Docker image.
- **hooks**: EventBus events this extension subscribes to, with priority
  (lower = runs first) and whether the hook is async.
- **tools**: LLM-callable tools this extension provides (see §5).
- **surfaces**: UI surfaces this extension registers (see §6).
- **settings**: User-configurable options, rendered as a form in the
  extension settings UI.

---

## 4. Browser Extensions (Tier 0)

### 4.1 Lifecycle

```
Install → Load → Init → Active → Disable → Unload
```

**Install**: Admin pushes manifest + JS to server, or user adds from
settings. Stored in `extensions` table with `tier = 'browser'`.

**Load**: On page load, after `events.js` but before `app.js`, the
extension loader injects `<script>` tags for all enabled browser extensions.
Load order respects declared dependencies.

**Init**: Each extension's entry script calls `Extensions.register()`:

```js
Extensions.register({
  id: 'cost-tracker',

  init(ctx) {
    // ctx.events  — scoped EventBus (only permitted events)
    // ctx.storage — scoped localStorage wrapper
    // ctx.settings — this extension's settings values
    // ctx.ui      — DOM injection points
    // ctx.api     — proxied fetch() to backend (auth injected)
    // ctx.model   — current model ID + resolved capabilities
    // ctx.user    — current user info (id, username, role)
    this.ctx = ctx;

    ctx.events.on('chat.message.send', (msg) => {
      const est = this.estimateTokens(msg.content);
      ctx.ui.inject('input-area', this.renderCost(est));
    });
  },

  destroy() {
    // Cleanup: remove DOM elements, unsubscribe events
  }
});
```

### 4.2 Context Object

The `ctx` object is the extension's API surface. It's scoped — an
extension that didn't declare `dom:sidebar` in permissions gets a `ctx.ui`
that throws on `inject('sidebar', ...)`.

```
ctx.events    — EventBus subscribe/publish (filtered by permissions)
ctx.storage   — localStorage namespace (extensions::{id}::*)
ctx.settings  — Read-only settings values from manifest
ctx.ui        — DOM injection into declared surfaces
ctx.api       — Proxied fetch() to backend (auth headers injected)
ctx.model     — Current model ID + resolved capabilities
ctx.user      — Current user info (id, username, role)
```

### 4.3 Admin-Pushed vs User-Installed

- **Admin-pushed**: `extensions` row with `is_system = true`. Loaded for
  all users. Users cannot disable.
- **User-installed**: Stored in user settings JSONB. Users can toggle.
- Both use the same runtime.

### 4.4 Security Model

Browser extensions run in the same origin. The security model is:

1. **Permission declaration** — extensions declare what they need; the
   loader enforces it via the `ctx` proxy.
2. **Admin review** — admin-pushed extensions are implicitly trusted.
   User-installed extensions get a "this extension can access: ..." prompt.
3. **Event scoping** — extensions only see events they declared in
   `permissions`.
4. **CSP headers** — strict Content-Security-Policy prevents inline script
   injection. Extension scripts are served from known paths only.

---

## 5. Browser-Defined Tools (The Bridge)

This is the critical innovation. The LLM tool call originates from the
provider response, arrives at the Go backend, and needs to execute in the
user's browser. The EventBus + WebSocket bridge makes this transparent.

### 5.1 The Flow

```
User sends message
    → Backend sends to LLM with tools[] from enabled extensions
    → LLM returns tool_call: { name: "read_file", args: {...} }
    → Backend checks tool registry → tool is tier:browser
    → Backend publishes via WebSocket:
        event: tool.call.{callId}
        data:  { tool: "read_file", args: {...}, callId: "uuid" }
    → Browser extension receives event, executes tool handler
    → Extension publishes result via WebSocket:
        event: tool.result.{callId}
        data:  { callId: "uuid", result: "file contents..." }
    → Backend receives result, feeds back to LLM as tool_result
    → LLM continues with the result
    → Response streams to user
```

### 5.2 Tool Registration

Extensions declare tools in their manifest and implement handlers:

```json
{
  "tools": [
    {
      "name": "estimate_cost",
      "description": "Estimate the token cost of a given text",
      "parameters": {
        "type": "object",
        "properties": {
          "text": { "type": "string", "description": "Text to estimate" }
        },
        "required": ["text"]
      },
      "tier": "browser"
    }
  ]
}
```

```js
Extensions.register({
  id: 'cost-tracker',
  init(ctx) {
    ctx.tools.handle('estimate_cost', async (args) => {
      const tokens = this.estimateTokens(args.text);
      return { tokens, estimated_cost: tokens * this.pricing / 1e6 };
    });
  }
});
```

### 5.3 Server-Side Tool Router

The completion handler maintains a tool registry. When building the
`tools[]` array for the LLM request:

```go
func (h *CompletionHandler) collectTools(userID string) []ToolSchema {
    var tools []ToolSchema

    // Tier 1: Starlark tools (server-side, execute inline)
    tools = append(tools, h.starlarkTools()...)

    // Tier 2: Sidecar tools (server-side, HTTP call)
    tools = append(tools, h.sidecarTools()...)

    // Tier 0: Browser tools (client-side, routed via WebSocket)
    if h.hub.IsConnected(userID) {
        tools = append(tools, h.browserTools(userID)...)
    }

    return tools
}
```

When a `tool_call` arrives and the tool is `tier: browser`:

```go
case "browser":
    callId := uuid.New().String()
    bus.Publish(events.Event{
        Type: "tool.call." + callId,
        Data: toolCallData,
        Room: "user:" + userID,
    })
    result, err := bus.WaitFor("tool.result." + callId, 30*time.Second)
```

### 5.4 Timeout and Fallback

Browser tools have a 30-second timeout. If the browser disconnects or the
tool fails, the backend sends a `tool_result` with an error message to the
LLM so it can recover gracefully.

### 5.5 Why This Matters

An extension author can write a tool in 20 lines of JavaScript that the
LLM can call. No Go code. No container. No deployment. The cluster manager
extension defines `kubectl_get`, `ceph_status`, `node_drain` as browser
tools. The editor extension defines `read_file`, `write_file`,
`search_replace`. The LLM doesn't know or care where the tool executes.

---

## 6. Extension Surfaces

*Shipped in v0.27.0.*

Extension surfaces are full-page custom UIs installed as `.surface`
archives and served at `/s/:slug`. They run inside the platform shell
with access to the authenticated API, theme system, and UI components.

See **[EXTENSION-SURFACES.md](EXTENSION-SURFACES.md)** for the complete
authoring guide including manifest format, platform API reference, CSS
custom properties, and a worked example.

### Summary

- **Archive format:** zip with `manifest.json` + `js/main.js` + optional `css/main.css`
- **Install:** Admin panel → Surfaces → Upload (no restart required)
- **Route:** `/s/{id}` — rendered via Go templates, DB lookup at request time
- **Entry point:** `js/main.js` mounts into `#extension-mount`
- **Platform access:** `API._get()`, `UI.toast()`, `Theme.resolved()`, `Events.on()`, `ChatPane.create()`
- **Theming:** CSS custom properties from `variables.css` (`--bg`, `--bg-surface`, `--text`, `--text-2`, `--accent`, etc.)
- **Admin API:** install, enable/disable, uninstall via `/api/v1/admin/surfaces/*`

### Relationship to Other Extension Types

Extension surfaces are the **most capable** extension type. They create
entirely new pages. Other extension types are lighter-weight:

| Type | Creates a page? | Uses |
|------|----------------|------|
| **Surface** (`/s/:slug`) | Yes — full page | Custom dashboards, tools, visualizations |
| **Block renderer** (Hook 3) | No — transforms message content | Mermaid diagrams, KaTeX math, CSV tables |
| **Tool** (Hook 5) | No — LLM-invocable function | Calculator, web search, file operations |
| **Post-render** (Hook 3) | No — enhances rendered messages | Syntax highlighting, link previews |

A single extension can combine multiple types — e.g. a surface for
configuration plus a tool for LLM interaction plus a renderer for
custom output formatting.

---

## 7. Extension Loader Architecture

### 7.1 Load Order

```html
<!-- Core -->
<script src="js/events.js"></script>
<script src="js/extensions.js"></script>   <!-- loader + registry -->

<!-- Extensions (injected by loader) -->
<script src="/api/v1/extensions/cost-tracker/assets/main.js"></script>
<script src="/api/v1/extensions/editor-mode/assets/main.js"></script>

<!-- App (runs after extensions registered) -->
<script src="js/api.js"></script>
<script src="js/ui.js"></script>
<script src="js/app.js"></script>
```

### 7.2 extensions.js (Core)

The extension loader and registry. ~200 lines. Responsibilities:

- Fetch enabled extensions from `/api/v1/extensions?tier=browser`
- Inject script tags in dependency order
- Provide `Extensions.register()` API
- Build scoped `ctx` objects per extension (permission enforcement)
- Manage surface activation/deactivation
- Collect browser tool schemas for the completion handler
- Route `tool.call.*` events to the correct handler

### 7.3 Backend Support

New tables (migration):

```sql
CREATE TABLE extensions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    ext_id VARCHAR(100) NOT NULL UNIQUE,      -- manifest id
    name VARCHAR(200) NOT NULL,
    tier VARCHAR(20) NOT NULL DEFAULT 'browser',
    manifest JSONB NOT NULL,
    assets_path TEXT,           -- filesystem path for browser JS
    endpoint TEXT,              -- URL for sidecar
    script TEXT,                -- inline Starlark source
    installed_by UUID REFERENCES users(id),
    is_system BOOLEAN DEFAULT false,
    is_enabled BOOLEAN DEFAULT true,
    scope VARCHAR(20) DEFAULT 'global',       -- global, team, personal
    team_id UUID REFERENCES teams(id),
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE extension_user_settings (
    extension_id UUID REFERENCES extensions(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    settings JSONB DEFAULT '{}',
    is_enabled BOOLEAN DEFAULT true,
    PRIMARY KEY (extension_id, user_id)
);
```

New endpoints:

```
GET    /api/v1/extensions                     — list enabled for current user
GET    /api/v1/extensions/:id/manifest        — get manifest
GET    /api/v1/extensions/:id/assets/*path    — serve browser JS
POST   /api/v1/extensions/:id/settings        — update user settings

POST   /api/v1/admin/extensions               — install extension
DELETE /api/v1/admin/extensions/:id            — uninstall
PUT    /api/v1/admin/extensions/:id            — enable/disable, config
```

---

## 8. EventBus Integration

The routing table from `events/types.go` expands:

```go
var Routes = map[string]Direction{
    // Core
    "chat.message.*":       DirBoth,
    "user.presence":        DirToClient,

    // Tool execution
    "tool.call.*":          DirToClient,    // Server → specific client
    "tool.result.*":        DirFromClient,  // Client → server

    // Surfaces
    "surface.activated":    DirLocal,       // Client-only
    "surface.deactivated":  DirLocal,

    // Extension lifecycle
    "extension.loaded":     DirLocal,
    "extension.error":      DirLocal,

    // Cross-extension (cluster manager example)
    "cluster.node.*":       DirLocal,       // Between browser extensions
    "cluster.alert.*":      DirBoth,        // Could notify server too
}
```

Browser extensions use `Events.on()` and `Events.emit()` — the same API
the core app uses. `DirLocal` events stay in the browser. `DirBoth` and
`DirFromClient` events cross the WebSocket.

---

## 9. Built-in Tools

These ship with core because multiple modes and services depend on them:

| Tool | Tier | Description |
|---|---|---|
| `web_search` | Sidecar | Search provider abstraction (SearXNG, Brave, DuckDuckGo) |
| `url_fetch` | Server | Retrieve and extract content from a URL |
| `note_create` | Server | Create a note with title, content, folder, tags |
| `note_search` | Server | Full-text search across user's notes |
| `note_update` | Server | Update note content |
| `note_list` | Server | List notes with optional folder filter |
| `kb_search` | Server | Semantic search across knowledge bases (future) |
| `task_create` | Server | Schedule a new task (future) |

Extension-provided tools (not core, expected early extensions):
- `read_file`, `write_file`, `search_replace` (Editor mode)
- `kubectl_get`, `ceph_status`, `node_ssh` (Cluster manager)
- `generate_image` (Image gen, browser or sidecar)
- `speak_text`, `transcribe_audio` (STT/TTS, browser via Web Speech API)

---

## 10. Design Principles

1. **Extensions are first-class.** A mode or feature implemented as an
   extension is indistinguishable from one built into core. No second-class
   citizens.

2. **The EventBus is the spine.** Extensions don't import each other.
   They publish and subscribe to events. This is how cluster manager
   extensions cooperate without knowing about each other at build time.

3. **Tools are location-transparent.** The LLM sees a tool schema. It
   doesn't know if the tool runs in the browser, in a Starlark sandbox,
   or in a container. The routing is the platform's problem.

4. **Permissions are declared, not discovered.** An extension says what it
   needs upfront. The loader enforces it. No ambient authority.

5. **The core stays small.** Auth, provider routing, message persistence,
   event bus, extension loader. That's core. Everything else is an
   extension — even if it ships with the project.

6. **Progressive capability.** A browser extension with zero tools and
   zero surfaces is just an EventBus subscriber. Add a tool and the LLM
   can call it. Add a surface and it becomes a mode. The same manifest
   format scales from "show token count" to "full IDE."

---

## Appendix A: Custom Renderers

Custom renderers are the simplest useful browser extension. They intercept
message content rendering to handle specific content types — no tools, no
surfaces, just a pattern match and a render function.

```js
Extensions.register({
  id: 'collapsible-code',
  init(ctx) {
    ctx.renderers.register('collapsible-code', {
      // Match fenced code blocks with >10 lines
      pattern: /^```(\w+)?\n([\s\S]{10,}?)```$/gm,
      render(match, container) {
        const lang = match[1] || 'text';
        const code = match[2];
        const lines = code.split('\n');
        const details = document.createElement('details');
        details.innerHTML = `
          <summary>${lang} — ${lines.length} lines</summary>
          <pre><code class="language-${lang}">${escapeHtml(code)}</code></pre>
        `;
        container.replaceWith(details);
      }
    });
  }
});
```

Expected first-party renderer extensions (ship with core or as official):

| Renderer | Matches | Renders |
|---|---|---|
| `collapsible-code` | Code blocks > N lines | `<details>` with expand/collapse |
| `html-preview` | ```html blocks | Sandboxed iframe with live preview |
| `mermaid` | ```mermaid blocks | SVG diagram via mermaid.js |
| `latex-math` | `$...$` and `$$...$$` | Rendered math via KaTeX |
| `doc-preview` | Generated HTML/PDF content | Preview panel with download link |

These are all ~30-50 line browser extensions. They don't need tool calling
or surfaces. They just pattern-match content in rendered messages.

---

## Appendix B: Model Roles

Extensions (and core services like compaction) may need to call LLMs for
their own purposes — summarization, embedding, classification — independent
of the user's selected chat model. **Model roles** are named slots that
resolve to a specific provider+model at runtime.

### B.1 Role Definition

```
┌──────────────────────────────────────────────────────────────┐
│  Role          │  Purpose                 │  Typical Model    │
├──────────────────────────────────────────────────────────────┤
│  utility       │  Summarization, routing  │  gpt-4o-mini      │
│                │  classification, triage  │  deepseek-chat     │
│  embedding     │  Vector generation for   │  text-embedding-3  │
│                │  KB search, note search  │  nomic-embed-text  │
└──────────────────────────────────────────────────────────────┘
```

> **Note:** The `generation` role (image/media) was removed in v0.10.2.
> Image generation will be extension-managed with its own provider config
> and API surface, not a named role slot.

### B.2 Configuration

Admin configures roles in global settings, with optional fallback chain:

```json
{
  "model_roles": {
    "utility": {
      "primary": { "provider_config_id": "abc", "model_id": "gpt-4o-mini" },
      "fallback": { "provider_config_id": "def", "model_id": "deepseek-chat" }
    },
    "embedding": {
      "primary": { "provider_config_id": "abc", "model_id": "text-embedding-3-small" },
      "fallback": null
    }
  }
}
```

Teams can override roles for their scope. Personal overrides TBD.

### B.3 Extension API

```js
// Browser extension requests a utility completion
const result = await ctx.models.complete('utility', {
  messages: [{ role: 'user', content: 'Summarize this in 2 sentences: ...' }]
});

// Server-side: extension requests an embedding
vec, err := models.Embed(ctx, "embedding", "text to embed")
```

The caller doesn't know which model or provider is being used. The role
system resolves it, tries fallback if primary fails, and logs usage
against the role for cost tracking.

### B.4 Core Consumers

| Consumer | Role | Purpose |
|---|---|---|
| Compaction service | `utility` | Summarize long conversations |
| Smart routing | `utility` | Classify intent, pick best model |
| Summarize & Continue | `utility` | Conversation compaction (v0.10.2) |
| Knowledge bases | `embedding` | Generate vectors for similarity search |
| Note search (future) | `embedding` | Semantic search across notes |
| Image gen extension | _(extension-managed)_ | Uses own provider config, not a role |

### B.5 Relationship to Smart Routing

Model roles and smart routing are complementary:

- **Model roles**: "I need *a* cheap model for summarization" → resolves
  to a specific provider+model based on admin config.
- **Smart routing**: "Route *this user's chat* to the best model" → policy
  engine that picks based on capabilities, cost, latency, team rules.

Roles are for background/system tasks. Routing is for user-facing chat.
Both use the same provider infrastructure and cost tracking.