core/EXTENSIONS.md

Chat Switchboard — Extension System Specification

Version: 0.2 Status: Design (pre-implementation) See also: ARCHITECTURE.md, ROADMAP.md

---

1. Philosophy

Chat Switchboard is a substrate, not an application. The core provides: authentication, provider routing, message persistence, an event bus, and a rendering surface. Everything else — editing, writing, cluster management, cost tracking, custom renderers — is an extension.

The goal is that someone with a problem and some JS (or Go, or Python) can solve it without forking the project. The "modes" — Editor, Article, Chat, Cluster Manager — are extensions that register surfaces, tools, and event handlers. This project doesn't have to build all of them. It has to make them possible.

---

2. Extension Tiers

	Tier 0: Browser	Tier 1: Starlark	Tier 2: Sidecar
Runs in	User's browser	Go server (embedded)	Separate container
Language	JavaScript	Starlark (Python subset)	Any
Deployed by	User or Admin push	Admin	Admin
Latency	Zero (client-side)	Low (in-process)	Network hop
Can access	DOM, EventBus, user context	Message data, DB reads (sandboxed)	Anything (HTTP, filesystem, network)
Cannot access	Server internals, other users' data	Network, filesystem, raw SQL	N/A (full access)
Trust model	Same-origin; user-scoped or admin-pushed	Starlark sandbox (no I/O)	Container isolation
Use cases	UI, rendering, shortcuts, client tools, modes	Routing rules, message transforms, logging	RAG, external APIs, webhooks, heavy compute

All three tiers communicate through the EventBus. A browser extension publishes tool.result.{callId} the same way a sidecar does — the bus doesn't care where the event originated.

Admin-pushed vs User-installed (Tier 0): Admin-pushed extensions load for all users (like a managed browser extension policy). User-installed extensions load from user settings and only affect that user's session. Both use the same runtime, same manifest format. The difference is governance, not execution.

---

3. Manifest Format

Every extension, regardless of tier, is described by a manifest:

{
  "id": "cost-tracker",
  "name": "Cost Tracker",
  "version": "1.0.0",
  "tier": "browser",
  "author": "jeff",
  "description": "Real-time token counting and cost estimation",

  "permissions": [
    "events:chat.message.*",
    "events:model.selected",
    "dom:input-area",
    "storage:local"
  ],

  "entry": "cost-tracker.js",

  "hooks": {
    "chat.message.send":    { "priority": 10, "async": false },
    "chat.message.received": { "priority": 50, "async": true }
  },

  "tools": [],
  "surfaces": [],

  "settings": {
    "showInline": {
      "type": "boolean",
      "label": "Show cost inline",
      "default": true
    }
  }
}

3.1 Fields

• id: Unique identifier. Namespaced by convention (jeff.cost-tracker).
• tier: browser | starlark | sidecar
• permissions: What the extension needs. The loader enforces these. Undeclared access is blocked (Tier 0 via proxy, Tier 1 via sandbox, Tier 2 via API scoping).
• entry: Browser: JS file path. Starlark: .star file. Sidecar: endpoint URL or Docker image.
• hooks: EventBus events this extension subscribes to, with priority (lower = runs first) and whether the hook is async.
• tools: LLM-callable tools this extension provides (see §5).
• surfaces: UI surfaces this extension registers (see §6).
• settings: User-configurable options, rendered as a form in the extension settings UI.

---

4. Browser Extensions (Tier 0)

4.1 Lifecycle

Install → Load → Init → Active → Disable → Unload

Install: Admin pushes manifest + JS to server, or user adds from settings. Stored in extensions table with tier = 'browser'.

Load: On page load, after events.js but before app.js, the extension loader injects <script> tags for all enabled browser extensions. Load order respects declared dependencies.

Init: Each extension's entry script calls Extensions.register():

Extensions.register({
  id: 'cost-tracker',

  init(ctx) {
    // ctx.events  — scoped EventBus (only permitted events)
    // ctx.storage — scoped localStorage wrapper
    // ctx.settings — this extension's settings values
    // ctx.ui      — DOM injection points
    // ctx.api     — proxied fetch() to backend (auth injected)
    // ctx.model   — current model ID + resolved capabilities
    // ctx.user    — current user info (id, username, role)
    this.ctx = ctx;

    ctx.events.on('chat.message.send', (msg) => {
      const est = this.estimateTokens(msg.content);
      ctx.ui.inject('input-area', this.renderCost(est));
    });
  },

  destroy() {
    // Cleanup: remove DOM elements, unsubscribe events
  }
});

4.2 Context Object

The ctx object is the extension's API surface. It's scoped — an extension that didn't declare dom:sidebar in permissions gets a ctx.ui that throws on inject('sidebar', ...).

ctx.events    — EventBus subscribe/publish (filtered by permissions)
ctx.storage   — localStorage namespace (extensions::{id}::*)
ctx.settings  — Read-only settings values from manifest
ctx.ui        — DOM injection into declared surfaces
ctx.api       — Proxied fetch() to backend (auth headers injected)
ctx.model     — Current model ID + resolved capabilities
ctx.user      — Current user info (id, username, role)

4.3 Admin-Pushed vs User-Installed

• Admin-pushed: extensions row with is_system = true. Loaded for all users. Users cannot disable.
• User-installed: Stored in user settings JSONB. Users can toggle.
• Both use the same runtime.

4.4 Security Model

Browser extensions run in the same origin. The security model is:

1. Permission declaration — extensions declare what they need; the loader enforces it via the ctx proxy.
2. Admin review — admin-pushed extensions are implicitly trusted. User-installed extensions get a "this extension can access: ..." prompt.
3. Event scoping — extensions only see events they declared in permissions.
4. CSP headers — strict Content-Security-Policy prevents inline script injection. Extension scripts are served from known paths only.

---

5. Browser-Defined Tools (The Bridge)

This is the critical innovation. The LLM tool call originates from the provider response, arrives at the Go backend, and needs to execute in the user's browser. The EventBus + WebSocket bridge makes this transparent.

5.1 The Flow

User sends message
    → Backend sends to LLM with tools[] from enabled extensions
    → LLM returns tool_call: { name: "read_file", args: {...} }
    → Backend checks tool registry → tool is tier:browser
    → Backend publishes via WebSocket:
        event: tool.call.{callId}
        data:  { tool: "read_file", args: {...}, callId: "uuid" }
    → Browser extension receives event, executes tool handler
    → Extension publishes result via WebSocket:
        event: tool.result.{callId}
        data:  { callId: "uuid", result: "file contents..." }
    → Backend receives result, feeds back to LLM as tool_result
    → LLM continues with the result
    → Response streams to user

5.2 Tool Registration

Extensions declare tools in their manifest and implement handlers:

{
  "tools": [
    {
      "name": "estimate_cost",
      "description": "Estimate the token cost of a given text",
      "parameters": {
        "type": "object",
        "properties": {
          "text": { "type": "string", "description": "Text to estimate" }
        },
        "required": ["text"]
      },
      "tier": "browser"
    }
  ]
}

Extensions.register({
  id: 'cost-tracker',
  init(ctx) {
    ctx.tools.handle('estimate_cost', async (args) => {
      const tokens = this.estimateTokens(args.text);
      return { tokens, estimated_cost: tokens * this.pricing / 1e6 };
    });
  }
});

5.3 Server-Side Tool Router

The completion handler maintains a tool registry. When building the tools[] array for the LLM request:

func (h *CompletionHandler) collectTools(userID string) []ToolSchema {
    var tools []ToolSchema

    // Tier 1: Starlark tools (server-side, execute inline)
    tools = append(tools, h.starlarkTools()...)

    // Tier 2: Sidecar tools (server-side, HTTP call)
    tools = append(tools, h.sidecarTools()...)

    // Tier 0: Browser tools (client-side, routed via WebSocket)
    if h.hub.IsConnected(userID) {
        tools = append(tools, h.browserTools(userID)...)
    }

    return tools
}

When a tool_call arrives and the tool is tier: browser:

case "browser":
    callId := uuid.New().String()
    bus.Publish(events.Event{
        Type: "tool.call." + callId,
        Data: toolCallData,
        Room: "user:" + userID,
    })
    result, err := bus.WaitFor("tool.result." + callId, 30*time.Second)

5.4 Timeout and Fallback

Browser tools have a 30-second timeout. If the browser disconnects or the tool fails, the backend sends a tool_result with an error message to the LLM so it can recover gracefully.

5.5 Why This Matters

An extension author can write a tool in 20 lines of JavaScript that the LLM can call. No Go code. No container. No deployment. The cluster manager extension defines kubectl_get, ceph_status, node_drain as browser tools. The editor extension defines read_file, write_file, search_replace. The LLM doesn't know or care where the tool executes.

---

6. Surfaces (Modes)

A "mode" is an extension that registers a surface — a UI region that replaces or augments the default chat area.

6.1 Surface Regions

┌─────────────────────────────────────────────┐
│  sidebar-top     │  surface-header           │
│                  │                           │
│  sidebar-nav     │                           │
│  (mode selector) │  surface-main             │
│                  │  (chat, editor, article,  │
│  sidebar-content │   cluster, ...)           │
│  (context panel) │                           │
│                  │                           │
│  sidebar-bottom  │  surface-footer           │
│                  │  (input area)             │
└─────────────────────────────────────────────┘

6.2 Surface Registration

{
  "surfaces": [
    {
      "id": "editor",
      "label": "Editor",
      "icon": "code",
      "regions": ["surface-main", "surface-footer", "sidebar-content"],
      "default": false
    }
  ]
}

Extensions.register({
  id: 'editor-mode',
  init(ctx) {
    ctx.surfaces.register('editor', {
      activate() {
        ctx.ui.replace('surface-main', this.renderEditor());
        ctx.ui.replace('surface-footer', this.renderEditorInput());
        ctx.ui.replace('sidebar-content', this.renderFileTree());
      },
      deactivate() {
        ctx.ui.restore('surface-main');
        ctx.ui.restore('surface-footer');
        ctx.ui.restore('sidebar-content');
      }
    });
  }
});

6.3 Mode Selector

When extensions register surfaces, a mode selector appears in the sidebar. Clicking a mode calls activate() on that surface and deactivate() on the current one. The bus event surface.activated fires so other extensions can react.

6.4 Core Surface

Chat mode is the default surface. It's not special architecturally — it's the surface that's active when no extension surface is selected. Pragmatically it stays in core because everything depends on it.

6.5 Planned Modes

Editor Mode (extension)

• File tree in sidebar, code editor in main, AI chat in split/overlay.
• Tools: read_file, write_file, search_replace, git_status, git_commit — browser tools backed by a git provider API.
• This is ai-editor rebuilt properly: tool calls are server-side, logged, token-counted, auditable.

Article Mode (extension)

• Outline in sidebar, rich text editor in main, AI assistant in panel.
• Tools: fetch_url, summarize_section, check_citation.
• The conversation is hidden; only the document matters.

Cluster Manager Mode (2-3 cooperating extensions)

• k8s-manager: kubectl tools, pod/deployment views.
• ceph-manager: ceph status, OSD management.
• node-manager: SSH commands, system metrics.
• These talk to each other via EventBus. When k8s-manager detects a node is NotReady, it publishes cluster.node.unhealthy. ceph-manager subscribes and checks OSD placement.
• The LLM sees ALL tools from ALL active cluster extensions. "Drain node-3, ensure Ceph rebalances, then cordon it" → kubectl_drain, ceph_osd_out, kubectl_cordon — each routed to the appropriate extension.

---

7. Extension Loader Architecture

7.1 Load Order

<!-- Core -->
<script src="js/events.js"></script>
<script src="js/extensions.js"></script>   <!-- loader + registry -->

<!-- Extensions (injected by loader) -->
<script src="/api/v1/extensions/cost-tracker/assets/main.js"></script>
<script src="/api/v1/extensions/editor-mode/assets/main.js"></script>

<!-- App (runs after extensions registered) -->
<script src="js/api.js"></script>
<script src="js/ui.js"></script>
<script src="js/app.js"></script>

7.2 extensions.js (Core)

The extension loader and registry. ~200 lines. Responsibilities:

• Fetch enabled extensions from /api/v1/extensions?tier=browser
• Inject script tags in dependency order
• Provide Extensions.register() API
• Build scoped ctx objects per extension (permission enforcement)
• Manage surface activation/deactivation
• Collect browser tool schemas for the completion handler
• Route tool.call.* events to the correct handler

7.3 Backend Support

New tables (migration):

CREATE TABLE extensions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    ext_id VARCHAR(100) NOT NULL UNIQUE,      -- manifest id
    name VARCHAR(200) NOT NULL,
    tier VARCHAR(20) NOT NULL DEFAULT 'browser',
    manifest JSONB NOT NULL,
    assets_path TEXT,           -- filesystem path for browser JS
    endpoint TEXT,              -- URL for sidecar
    script TEXT,                -- inline Starlark source
    installed_by UUID REFERENCES users(id),
    is_system BOOLEAN DEFAULT false,
    is_enabled BOOLEAN DEFAULT true,
    scope VARCHAR(20) DEFAULT 'global',       -- global, team, personal
    team_id UUID REFERENCES teams(id),
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE extension_user_settings (
    extension_id UUID REFERENCES extensions(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    settings JSONB DEFAULT '{}',
    is_enabled BOOLEAN DEFAULT true,
    PRIMARY KEY (extension_id, user_id)
);

New endpoints:

GET    /api/v1/extensions                     — list enabled for current user
GET    /api/v1/extensions/:id/manifest        — get manifest
GET    /api/v1/extensions/:id/assets/*path    — serve browser JS
POST   /api/v1/extensions/:id/settings        — update user settings

POST   /api/v1/admin/extensions               — install extension
DELETE /api/v1/admin/extensions/:id            — uninstall
PUT    /api/v1/admin/extensions/:id            — enable/disable, config

---

8. EventBus Integration

The routing table from events/types.go expands:

var Routes = map[string]Direction{
    // Core
    "chat.message.*":       DirBoth,
    "user.presence":        DirToClient,

    // Tool execution
    "tool.call.*":          DirToClient,    // Server → specific client
    "tool.result.*":        DirFromClient,  // Client → server

    // Surfaces
    "surface.activated":    DirLocal,       // Client-only
    "surface.deactivated":  DirLocal,

    // Extension lifecycle
    "extension.loaded":     DirLocal,
    "extension.error":      DirLocal,

    // Cross-extension (cluster manager example)
    "cluster.node.*":       DirLocal,       // Between browser extensions
    "cluster.alert.*":      DirBoth,        // Could notify server too
}

Browser extensions use Events.on() and Events.emit() — the same API the core app uses. DirLocal events stay in the browser. DirBoth and DirFromClient events cross the WebSocket.

---

9. Built-in Tools

These ship with core because multiple modes and services depend on them:

Tool	Tier	Description
web_search	Sidecar	Search provider abstraction (SearXNG, Brave, DuckDuckGo)
url_fetch	Server	Retrieve and extract content from a URL
note_create	Server	Create a note with title, content, folder, tags
note_search	Server	Full-text search across user's notes
note_update	Server	Update note content
note_list	Server	List notes with optional folder filter
kb_search	Server	Semantic search across knowledge bases (future)
task_create	Server	Schedule a new task (future)

Extension-provided tools (not core, expected early extensions):

• read_file, write_file, search_replace (Editor mode)
• kubectl_get, ceph_status, node_ssh (Cluster manager)
• generate_image (Image gen, browser or sidecar)
• speak_text, transcribe_audio (STT/TTS, browser via Web Speech API)

---

10. Design Principles

1. Extensions are first-class. A mode or feature implemented as an extension is indistinguishable from one built into core. No second-class citizens.

2. The EventBus is the spine. Extensions don't import each other. They publish and subscribe to events. This is how cluster manager extensions cooperate without knowing about each other at build time.

3. Tools are location-transparent. The LLM sees a tool schema. It doesn't know if the tool runs in the browser, in a Starlark sandbox, or in a container. The routing is the platform's problem.

4. Permissions are declared, not discovered. An extension says what it needs upfront. The loader enforces it. No ambient authority.

5. The core stays small. Auth, provider routing, message persistence, event bus, extension loader. That's core. Everything else is an extension — even if it ships with the project.

6. Progressive capability. A browser extension with zero tools and zero surfaces is just an EventBus subscriber. Add a tool and the LLM can call it. Add a surface and it becomes a mode. The same manifest format scales from "show token count" to "full IDE."

---

Appendix A: Custom Renderers

Custom renderers are the simplest useful browser extension. They intercept message content rendering to handle specific content types — no tools, no surfaces, just a pattern match and a render function.

Extensions.register({
  id: 'collapsible-code',
  init(ctx) {
    ctx.renderers.register('collapsible-code', {
      // Match fenced code blocks with >10 lines
      pattern: /^```(\w+)?\n([\s\S]{10,}?)```$/gm,
      render(match, container) {
        const lang = match[1] || 'text';
        const code = match[2];
        const lines = code.split('\n');
        const details = document.createElement('details');
        details.innerHTML = `
          <summary>${lang} — ${lines.length} lines</summary>
          <pre><code class="language-${lang}">${escapeHtml(code)}</code></pre>
        `;
        container.replaceWith(details);
      }
    });
  }
});

Expected first-party renderer extensions (ship with core or as official):

Renderer	Matches	Renders
collapsible-code	Code blocks > N lines	<details> with expand/collapse
html-preview	```html blocks	Sandboxed iframe with live preview
mermaid	```mermaid blocks	SVG diagram via mermaid.js
latex-math	$...$ and $$...$$	Rendered math via KaTeX
doc-preview	Generated HTML/PDF content	Preview panel with download link

These are all ~30-50 line browser extensions. They don't need tool calling or surfaces. They just pattern-match content in rendered messages.

---

Appendix B: Model Roles

Extensions (and core services like compaction) may need to call LLMs for their own purposes — summarization, embedding, classification — independent of the user's selected chat model. Model roles are named slots that resolve to a specific provider+model at runtime.

B.1 Role Definition

┌──────────────────────────────────────────────────────────────┐
│  Role          │  Purpose                 │  Typical Model    │
├──────────────────────────────────────────────────────────────┤
│  utility       │  Summarization, routing  │  gpt-4o-mini      │
│                │  classification, triage  │  deepseek-chat     │
│  embedding     │  Vector generation for   │  text-embedding-3  │
│                │  KB search, note search  │  nomic-embed-text  │
│  generation    │  Image/media generation  │  dall-e-3          │
│                │                          │  stable-diffusion  │
└──────────────────────────────────────────────────────────────┘

B.2 Configuration

Admin configures roles in global settings, with optional fallback chain:

{
  "model_roles": {
    "utility": {
      "primary": { "provider_config_id": "abc", "model_id": "gpt-4o-mini" },
      "fallback": { "provider_config_id": "def", "model_id": "deepseek-chat" }
    },
    "embedding": {
      "primary": { "provider_config_id": "abc", "model_id": "text-embedding-3-small" },
      "fallback": null
    }
  }
}

Teams can override roles for their scope. Personal overrides TBD.

B.3 Extension API

// Browser extension requests a utility completion
const result = await ctx.models.complete('utility', {
  messages: [{ role: 'user', content: 'Summarize this in 2 sentences: ...' }]
});

// Server-side: extension requests an embedding
vec, err := models.Embed(ctx, "embedding", "text to embed")

The caller doesn't know which model or provider is being used. The role system resolves it, tries fallback if primary fails, and logs usage against the role for cost tracking.

B.4 Core Consumers

Consumer	Role	Purpose
Compaction service	utility	Summarize long conversations
Smart routing	utility	Classify intent, pick best model
Knowledge bases	embedding	Generate vectors for similarity search
Note search (future)	embedding	Semantic search across notes
Image gen extension	generation	Create images from descriptions

B.5 Relationship to Smart Routing

Model roles and smart routing are complementary:

• Model roles: "I need a cheap model for summarization" → resolves to a specific provider+model based on admin config.
• Smart routing: "Route this user's chat to the best model" → policy engine that picks based on capabilities, cost, latency, team rules.

Roles are for background/system tasks. Routing is for user-facing chat. Both use the same provider infrastructure and cost tracking.