core/EXTENSIONS.md

Chat Switchboard — Extension System Specification

Version: 0.1 draft Status: Design Applies to: v0.7.x+ Companion to: ARCHITECTURE.md (core services + terminology)

---

1. Philosophy

Chat Switchboard is a substrate, not an application. The core provides: authentication, provider routing, message persistence, an event bus, and a rendering surface. Everything else — editing, writing, cluster management, cost tracking, custom renderers — is an extension.

The goal is that someone with a problem and some JS (or Go, or Python) can solve it without forking the project. The "modes" Jeff envisions — Editor, Article, Chat, Cluster Manager — are just extensions that register surfaces, tools, and event handlers. This project doesn't have to build all of them. It just has to make them possible.

---

2. Extension Tiers

	Tier 0: Browser	Tier 1: Starlark	Tier 2: Sidecar
Runs in	User's browser	Go server (embedded)	Separate container
Language	JavaScript	Starlark (Python subset)	Any
Deployed by	User or Admin push	Admin	Admin
Latency	Zero (client-side)	Low (in-process)	Network hop
Can access	DOM, EventBus, LocalStorage, user context	Message data, DB reads (sandboxed)	Anything (HTTP, filesystem, network)
Cannot access	Server internals, other users' data	Network, filesystem, raw SQL	N/A (full access)
Trust model	Same-origin; user-scoped or admin-pushed	Starlark sandbox (no I/O)	Container isolation
Use cases	UI, rendering, shortcuts, client tools, modes	Routing rules, message transforms, logging	RAG, external APIs, webhooks, heavy compute

All three tiers communicate through the EventBus. A browser extension publishes tool.result.{callId} the same way a sidecar does — the bus doesn't care where the event originated.

---

3. Manifest Format

Every extension, regardless of tier, is described by a manifest:

{
  "id": "cost-tracker",
  "name": "Cost Tracker",
  "version": "1.0.0",
  "tier": "browser",
  "author": "jeff",
  "description": "Real-time token counting and cost estimation",

  "permissions": [
    "events:chat.message.*",
    "events:model.selected",
    "dom:input-area",
    "storage:local"
  ],

  "entry": "cost-tracker.js",

  "hooks": {
    "chat.message.send":    { "priority": 10, "async": false },
    "chat.message.received": { "priority": 50, "async": true }
  },

  "tools": [],

  "surfaces": [],

  "settings": {
    "showInline": {
      "type": "boolean",
      "label": "Show cost inline",
      "default": true
    }
  }
}

3.1 Fields

• id: Unique identifier. Namespaced by convention (jeff.cost-tracker).
• tier: browser | starlark | sidecar
• permissions: What the extension needs access to. The loader enforces these. Undeclared access is blocked (Tier 0 via proxy, Tier 1 via sandbox, Tier 2 via API scoping).
• entry: For browser: JS file path. For starlark: .star file. For sidecar: endpoint URL or Docker image.
• hooks: EventBus events this extension subscribes to, with priority (lower = runs first) and whether the hook is async.
• tools: LLM-callable tools this extension provides (see §5).
• surfaces: UI surfaces this extension registers (see §6).
• settings: User-configurable options, rendered as a form in the extension settings UI.

---

4. Browser Extensions (Tier 0)

4.1 Lifecycle

Install → Load → Init → Active → Disable → Unload

Install: Admin pushes manifest + JS to server, or user adds from settings. Stored in extensions table with tier = 'browser'.

Load: On page load, after events.js but before app.js, the extension loader injects <script> tags for all enabled browser extensions. Load order respects declared dependencies.

Init: Each extension's entry script calls Extensions.register():

Extensions.register({
  id: 'cost-tracker',

  init(ctx) {
    // ctx.events  — scoped EventBus (only permitted events)
    // ctx.storage — scoped localStorage wrapper
    // ctx.settings — this extension's settings values
    // ctx.ui      — DOM injection points
    this.ctx = ctx;

    ctx.events.on('chat.message.send', (msg) => {
      const est = this.estimateTokens(msg.content);
      ctx.ui.inject('input-area', this.renderCost(est));
    });

    ctx.events.on('model.selected', ({ capabilities }) => {
      this.pricing = this.lookupPricing(capabilities);
    });
  },

  destroy() {
    // Cleanup: remove DOM elements, unsubscribe events
  },

  estimateTokens(text) { /* ... */ },
  renderCost(est) { /* ... */ },
  lookupPricing(caps) { /* ... */ }
});

4.2 Context Object

The ctx object is the extension's API surface. It's scoped — an extension that didn't declare dom:sidebar in permissions gets a ctx.ui that throws on inject('sidebar', ...).

ctx.events    — EventBus subscribe/publish (filtered by permissions)
ctx.storage   — localStorage namespace (extensions::{id}::*)
ctx.settings  — Read-only settings values from manifest
ctx.ui        — DOM injection into declared surfaces
ctx.api       — Proxied fetch() to backend (auth headers injected)
ctx.model     — Current model ID + resolved capabilities
ctx.user      — Current user info (id, username, role)

4.3 Admin-Pushed vs User-Installed

• Admin-pushed: extensions table row with is_system = true. Loaded for all users. Users cannot disable. JS served from /api/v1/extensions/{id}/assets/.
• User-installed: Stored in user settings JSONB. Users can enable/disable. JS loaded from same asset path or inline (for small scripts).
• Both use the same runtime. The difference is governance, not execution.

4.4 Security Model

Browser extensions run in the same origin. They can't be truly sandboxed without iframes (which breaks DOM injection). The security model is:

1. Permission declaration — extensions declare what they need; the loader enforces it via the ctx proxy.
2. Admin review — admin-pushed extensions are implicitly trusted. User-installed extensions get a "this extension can access: ..." prompt.
3. Event scoping — extensions only see events they declared in permissions. The scoped EventBus filters unpermitted subscriptions.
4. CSP headers — strict Content-Security-Policy prevents inline script injection. Extension scripts are served from known paths only.

---

5. Browser-Defined Tools (The Bridge)

This is the critical innovation. ai-editor proved that LLM tools work in browser JS. But in Chat Switchboard, the completion handler runs server-side. The tool call originates from the LLM response, arrives at the Go backend, and needs to execute in the user's browser.

5.1 The Flow

User sends message
    → Backend sends to LLM with tools[] from enabled extensions
    → LLM returns tool_call: { name: "read_file", args: {...} }
    → Backend sees tool_call, checks tool registry
    → Tool is tier:browser → Backend publishes via WebSocket:
        event: tool.call.{callId}
        data:  { tool: "read_file", args: {...}, callId: "uuid" }
    → Browser extension receives event, executes tool
    → Extension publishes result via WebSocket:
        event: tool.result.{callId}
        data:  { callId: "uuid", result: "file contents..." }
    → Backend receives result, feeds back to LLM as tool_result message
    → LLM continues with the result
    → Response streams to user

5.2 Tool Registration

Extensions declare tools in their manifest and implement them:

{
  "tools": [
    {
      "name": "estimate_cost",
      "description": "Estimate the token cost of a given text",
      "parameters": {
        "type": "object",
        "properties": {
          "text": { "type": "string", "description": "Text to estimate" }
        },
        "required": ["text"]
      },
      "tier": "browser"
    }
  ]
}

Extensions.register({
  id: 'cost-tracker',
  init(ctx) {
    ctx.tools.handle('estimate_cost', async (args) => {
      const tokens = this.estimateTokens(args.text);
      return { tokens, estimated_cost_usd: tokens * this.pricing.outputPerM / 1e6 };
    });
  }
});

5.3 Server-Side Tool Router

The completion handler maintains a tool registry. When building the tools[] array for the LLM request:

func (h *CompletionHandler) collectTools(userID string) []Tool {
    var tools []Tool

    // Tier 1: Starlark tools (server-side, execute inline)
    tools = append(tools, h.starlarkTools()...)

    // Tier 2: Sidecar tools (server-side, HTTP call)
    tools = append(tools, h.sidecarTools()...)

    // Tier 0: Browser tools (client-side, routed via WebSocket)
    // Only included if user has WebSocket connected
    if hub.IsConnected(userID) {
        tools = append(tools, h.browserTools(userID)...)
    }

    return tools
}

When a tool_call arrives and the tool is tier: browser:

case "browser":
    callId := uuid.New().String()
    // Publish to user's WebSocket
    bus.Publish(events.Event{
        Type: "tool.call." + callId,
        Data: toolCallData,
        Room: "user:" + userID,
    })
    // Wait for result (with timeout)
    result, err := bus.WaitFor("tool.result." + callId, 30*time.Second)

5.4 Timeout and Fallback

Browser tools have a 30-second timeout. If the user's browser disconnects or the tool fails, the backend sends a tool_result with an error message to the LLM so it can recover gracefully. This is no different from how ai-editor handles tool failures — the LLM gets an error and adapts.

5.5 Why This Matters

This means an extension author can write a tool in 20 lines of JavaScript that the LLM can call. No Go code. No container. No deployment. The cluster manager extension defines kubectl_get, ceph_status, node_drain as browser tools. The editor extension defines read_file, write_file, search_replace. The article extension defines fetch_source, check_citation.

The LLM doesn't know or care where the tool executes. It sees a tool schema, calls it, gets a result. The bus handles the routing.

---

6. Surfaces (Modes)

A "mode" is an extension that registers a surface — a UI region that replaces or augments the default chat area. The core app provides injection points:

┌─────────────────────────────────────────────┐
│  sidebar-top     │  surface-header           │
│                  │                           │
│  sidebar-nav     │                           │
│  (mode selector) │  surface-main             │
│                  │  (chat, editor, article,  │
│  sidebar-content │   cluster, ...)           │
│  (context panel) │                           │
│                  │                           │
│  sidebar-bottom  │  surface-footer           │
│                  │  (input area)             │
└─────────────────────────────────────────────┘

6.1 Surface Registration

{
  "surfaces": [
    {
      "id": "editor",
      "label": "Editor",
      "icon": "code",
      "regions": ["surface-main", "surface-footer", "sidebar-content"],
      "default": false
    }
  ]
}

Extensions.register({
  id: 'editor-mode',
  init(ctx) {
    ctx.surfaces.register('editor', {
      activate() {
        // Replace surface-main with editor UI
        ctx.ui.replace('surface-main', this.renderEditor());
        ctx.ui.replace('surface-footer', this.renderEditorInput());
        ctx.ui.replace('sidebar-content', this.renderFileTree());
      },
      deactivate() {
        // Restore defaults
        ctx.ui.restore('surface-main');
        ctx.ui.restore('surface-footer');
        ctx.ui.restore('sidebar-content');
      }
    });
  }
});

6.2 The Core Surfaces

Chat mode is just the default surface. It's not special — it's the surface that's active when no extension surface is selected. In theory, even chat mode could be extracted into an extension, but pragmatically it stays in core because everything depends on it.

6.3 Mode Selector

When extensions register surfaces, a mode selector appears in the sidebar (below the brand, above chat history). Clicking a mode calls activate() on that surface and deactivate() on the current one.

The bus event surface.activated fires so other extensions can react. The cost tracker might show different metrics in editor mode vs chat mode.

6.4 Jeff's Planned Modes

Chat Mode (core)

• What exists today, plus tool calling and richer message types.
• The "default surface" that everything else builds on.

Editor Mode (extension)

• Surfaces: file tree in sidebar, code editor in main, AI chat in a split pane or overlay.
• Tools: read_file, write_file, search_replace, run_command, git_status, git_commit — all browser tools backed by a git provider API (GitHub, Gitea).
• This is ai-editor rebuilt properly: the tool definitions are JS, the LLM calls happen through the standard completion handler, and the file operations go through a git provider abstraction.
• The key difference from ai-editor: the completion handler is server-side, so tool calls are logged, token-counted, and auditable.

Article Mode (extension)

• Surfaces: outline in sidebar, rich text editor in main, AI assistant in a panel.
• Tools: fetch_url, summarize_section, check_citation, suggest_structure.
• Think: a writing environment where the AI is a research assistant, not a chatbot. The conversation is hidden; only the document matters.

Cluster Manager Mode (extension — or extensions plural)

• This one is interesting because it's actually 2-3 cooperating extensions:
  • k8s-manager: tools for kubectl operations, surfaces for pod/deployment views.
  • ceph-manager: tools for ceph status, OSD management, pool operations.
  • node-manager: tools for SSH commands, system metrics, package management.
• These extensions talk to each other via the EventBus. When k8s-manager detects a node is NotReady, it publishes cluster.node.unhealthy. node-manager subscribes and surfaces diagnostics. ceph-manager subscribes and checks if that node had OSDs.
• The LLM sees ALL the tools from ALL active cluster extensions. "Drain node-3, ensure Ceph rebalances, then cordon it" becomes a multi-tool conversation where the LLM calls kubectl_drain, then ceph_osd_out, then kubectl_cordon — each routed to the appropriate extension.
• This is what you do manually with Claude Desktop today. The extension system makes it one coherent interface.

---

7. Extension Loader Architecture

7.1 Load Order

<!-- Core -->
<script src="js/events.js"></script>
<script src="js/extensions.js"></script>   <!-- NEW: loader + registry -->

<!-- Extensions (injected by loader) -->
<script src="/api/v1/extensions/cost-tracker/assets/main.js"></script>
<script src="/api/v1/extensions/editor-mode/assets/main.js"></script>

<!-- App (runs after extensions registered) -->
<script src="js/api.js"></script>
<script src="js/ui.js"></script>
<script src="js/app.js"></script>

7.2 extensions.js (Core)

The extension loader and registry. ~200 lines. Responsibilities:

• Fetch enabled extensions list from /api/v1/extensions?tier=browser
• Inject script tags in dependency order
• Provide Extensions.register() API
• Build scoped ctx objects per extension (permission enforcement)
• Manage surface activation/deactivation
• Collect browser tool schemas for the completion handler
• Route tool.call.* events to the correct handler
• Provide Extensions.list(), Extensions.get(), Extensions.settings()

7.3 Backend Support

New tables (migration 006):

CREATE TABLE extensions (
    -- already exists from 001, but needs columns:
    tier VARCHAR(20) NOT NULL DEFAULT 'browser',  -- browser, starlark, sidecar
    manifest JSONB NOT NULL,
    assets_path TEXT,        -- filesystem path for browser JS
    endpoint TEXT,           -- URL for sidecar
    script TEXT,             -- inline Starlark source
    installed_by UUID REFERENCES users(id),
    is_system BOOLEAN DEFAULT false
);

CREATE TABLE extension_user_settings (
    extension_id UUID REFERENCES extensions(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    settings JSONB DEFAULT '{}',
    is_enabled BOOLEAN DEFAULT true,
    PRIMARY KEY (extension_id, user_id)
);

New endpoints:

GET    /api/v1/extensions                     — list enabled for current user
GET    /api/v1/extensions/:id/manifest        — get manifest
GET    /api/v1/extensions/:id/assets/*path    — serve browser JS
POST   /api/v1/extensions/:id/settings        — update user settings

POST   /api/v1/admin/extensions               — install extension
DELETE /api/v1/admin/extensions/:id            — uninstall
PUT    /api/v1/admin/extensions/:id            — update (enable/disable, config)

7.4 Tool Registry Integration

The completion handler's tool collection expands:

func (h *CompletionHandler) collectTools(userID string, configID string) []ToolSchema {
    var tools []ToolSchema

    // 1. Model-native tools (if provider supports them)
    caps := h.getModelCapabilities(model, configID)
    if !caps.ToolCalling {
        return nil  // Model doesn't support tools, skip everything
    }

    // 2. Server-side extension tools (Starlark + Sidecar)
    tools = append(tools, h.serverTools(userID)...)

    // 3. Browser tools (only if WebSocket connected)
    if h.hub.IsConnected(userID) {
        tools = append(tools, h.browserTools(userID)...)
    }

    return tools
}

---

8. EventBus Integration

The routing table from events/types.go expands:

var Routes = map[string]Direction{
    // Core
    "chat.message.*":       DirBoth,
    "user.presence":        DirToClient,

    // Tool execution
    "tool.call.*":          DirToClient,    // Server → specific client
    "tool.result.*":        DirFromClient,  // Client → server

    // Surfaces
    "surface.activated":    DirLocal,       // Client-only
    "surface.deactivated":  DirLocal,

    // Extension lifecycle
    "extension.loaded":     DirLocal,
    "extension.error":      DirLocal,

    // Cross-extension (cluster manager example)
    "cluster.node.*":       DirLocal,       // Between browser extensions
    "cluster.alert.*":      DirBoth,        // Could notify server too
}

Browser extensions use Events.on() and Events.emit() — the same API the core app uses. Events with DirBoth or DirFromClient cross the WebSocket. DirLocal events stay in the browser. This means cluster manager extensions can coordinate locally at zero latency, and only publish to the server when something needs persistence or notification.

---

9. Implementation Roadmap

Phase A: Foundation

• extensions.js — loader, registry, scoped context
• Extensions.register() API with permission enforcement
• Manifest format parser and validator
• Admin endpoints for extension management
• Asset serving endpoint
• Extension settings UI in Settings modal

Phase B: Browser Tools

• ctx.tools.handle() API for browser tool registration
• Tool schema collection in completion handler
• tool.call.* / tool.result.* WebSocket routing
• Timeout and error handling for browser tools
• Tool execution in EventBus WaitFor pattern
• Tool use display in chat messages

Phase C: Surfaces

• Surface registration and activation API
• Mode selector in sidebar
• ctx.ui.replace() / ctx.ui.restore() for region management
• surface.activated / surface.deactivated events

Phase D: First Extensions

• Cost tracker (browser, proof of concept)
• Slash commands (browser, message transforms)
• Custom renderers (browser, mermaid/latex/etc)
• Editor mode (browser, with git provider tools)

Phase E: Server-Side Tiers

• Starlark runtime integration
• Sidecar HTTP tool protocol
• Server-side tool execution in completion handler
• Web search as sidecar extension

---

10. Design Principles

1. Extensions are first-class. The system is designed so that a mode or feature implemented as an extension is indistinguishable from one built into core. No second-class citizens.

2. The EventBus is the spine. Extensions don't import each other. They publish and subscribe to events. This is how the cluster manager extensions cooperate without knowing about each other at build time.

3. Tools are location-transparent. The LLM sees a tool schema. It doesn't know if the tool runs in the browser, in a Starlark sandbox, or in a container in another data center. The routing is the platform's problem.

4. Permissions are declared, not discovered. An extension says what it needs upfront. The loader enforces it. No ambient authority.

5. The core stays small. Auth, provider routing, message persistence, event bus, extension loader. That's core. Everything else is an extension — even if it ships with the project.

6. Progressive capability. A browser extension with zero tools and zero surfaces is just an EventBus subscriber. Add a tool and the LLM can call it. Add a surface and it becomes a mode. The same manifest format scales from "show token count" to "full IDE."