core/docs/DESIGN-CM6.md

# DESIGN: CodeMirror 6 Integration

**Version:** v0.18.0 (or fits into current cycle)
**Status:** Draft
**Scope:** Add CM6 as a vendored dependency, bundled at Docker build time via esbuild. Three integration surfaces.

---

## Motivation

Three features converge on the same dependency:

1. **Chat input** — live markdown rendering (backtick → code block, bold, etc.)
2. **Extension editor** — syntax-highlighted JavaScript/JSON editing in admin panel (replaces bare `<textarea>`)
3. **Code editing surface** (roadmap) — future extension surface type for code review, snippets, etc.

CM6 is ESM-native and expects a bundler. The build step is contained in the Docker image build — the frontend continues to ship as static assets served by nginx. No runtime build tooling, no change to the developer experience for non-CM6 code.

---

## Architecture

### Build Pipeline

New Docker stage between the existing `vendor` stage and the final `nginx` stage:

```
Stage 1: vendor       (existing — marked, purify, mermaid, katex)
Stage 2: cm6-build    (NEW — npm install + esbuild → single IIFE bundle)
Stage 3: nginx        (existing — copies src/ + vendor/ + cm6 bundle)
```

The CM6 build stage:

```dockerfile
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/package.json src/editor/build.mjs ./
RUN npm install --production=false
RUN node build.mjs
# Output: /build/dist/codemirror.bundle.js (~180-250KB minified)
#         /build/dist/codemirror.bundle.css (~5-10KB)
```

Final stage addition:

```dockerfile
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
```

### Bundle Structure

A single esbuild entrypoint (`src/editor/index.mjs`) that imports CM6 packages and exposes factory functions on `window.CM`:

```javascript
// src/editor/index.mjs — esbuild entrypoint
import { EditorView, keymap, placeholder, ViewPlugin, ... } from '@codemirror/view';
import { EditorState, ... } from '@codemirror/state';
import { markdown, markdownLanguage } from '@codemirror/lang-markdown';
import { javascript } from '@codemirror/lang-javascript';
import { json } from '@codemirror/lang-json';
import { sql } from '@codemirror/lang-sql';
import { html } from '@codemirror/lang-html';
import { css } from '@codemirror/lang-css';
import { yaml } from '@codemirror/lang-yaml';
import { go } from '@codemirror/lang-go';
import { python } from '@codemirror/lang-python';
import { rust } from '@codemirror/lang-rust';
import { oneDark } from '@codemirror/theme-one-dark';
import { defaultKeymap, indentWithTab } from '@codemirror/commands';
import { bracketMatching, ... } from '@codemirror/language';
import { highlightSelectionMatches, searchKeymap } from '@codemirror/search';
import { autocompletion } from '@codemirror/autocomplete';
import { vim } from '@replit/codemirror-vim';
import { emacs } from '@replit/codemirror-emacs';

// Expose on window for script-tag consumption
window.CM = {
    EditorView,
    EditorState,

    // Factory: chat input (markdown mode, minimal chrome)
    chatInput(target, opts = {}) { ... },

    // Factory: code editor (full features, language auto-detect)
    codeEditor(target, opts = {}) { ... },

    // Supported languages for code editor
    languages: { markdown, javascript, json, sql, html, css, yaml, go, python, rust },

    // Keybinding modes (applied via user preference)
    keybindings: { vim, emacs },
};
```

### esbuild Config

```javascript
// src/editor/build.mjs
import { build } from 'esbuild';

await build({
    entryPoints: ['index.mjs'],
    bundle: true,
    minify: true,
    format: 'iife',
    target: ['es2020'],
    outfile: 'dist/codemirror.bundle.js',
    // CSS extracted automatically by esbuild
});
```

### package.json (editor only)

```json
{
  "private": true,
  "type": "module",
  "devDependencies": {
    "esbuild": "^0.25.0"
  },
  "dependencies": {
    "@codemirror/autocomplete": "^6",
    "@codemirror/commands": "^6",
    "@codemirror/lang-css": "^6",
    "@codemirror/lang-go": "^6",
    "@codemirror/lang-html": "^6",
    "@codemirror/lang-javascript": "^6",
    "@codemirror/lang-json": "^6",
    "@codemirror/lang-markdown": "^6",
    "@codemirror/lang-sql": "^6",
    "@codemirror/lang-python": "^6",
    "@codemirror/lang-rust": "^6",
    "@codemirror/lang-yaml": "^6",
    "@codemirror/language": "^6",
    "@codemirror/search": "^6",
    "@codemirror/state": "^6",
    "@codemirror/theme-one-dark": "^6",
    "@codemirror/view": "^6",
    "@replit/codemirror-vim": "^6",
    "@replit/codemirror-emacs": "^6"
  }
}
```

---

## Integration Surfaces

### 1. Chat Input — Markdown Mode

Replace `<textarea id="messageInput">` with a CM6 instance configured for chat composition.

**Behavior:**

- Markdown syntax highlighting as you type (fenced code blocks, bold, italic, links, etc.)
- Shift+Enter inserts newline (existing behavior preserved)
- Enter sends message (existing behavior preserved)
- Auto-growing height (CM6 handles this natively via `EditorView.contentAttributes`)
- Placeholder text: "Send a message..."
- No line numbers, no gutter — minimal chrome
- Tab inserts real tab (indentWithTab) inside code blocks; normal behavior outside

**Factory:**

```javascript
CM.chatInput(document.getElementById('messageInputContainer'), {
    placeholder: 'Send a message...',
    onSubmit: (text) => sendMessage(text),        // Enter key
    onChange: (text) => updateInputTokens(text),   // live token count
    darkMode: true,
});
```

**Migration path:**

The existing code reads `document.getElementById('messageInput').value` in many places. The factory returns an object with a `.getValue()` / `.setValue()` / `.focus()` API that matches the textarea interface. A thin shim keeps all existing callsites working:

```javascript
// After CM.chatInput() creates the editor:
const editor = CM.chatInput(...);

// Shim: existing code that reads .value still works
Object.defineProperty(document.getElementById('messageInput'), 'value', {
    get: () => editor.getValue(),
    set: (v) => editor.setValue(v),
});
```

Alternatively (cleaner): introduce `ChatInput.getValue()` / `ChatInput.setValue()` / `ChatInput.focus()` and update callsites. There aren't many — `chat.js`, `attachments.js`, `tokens.js`, and a few event handlers.

### 2. Extension Editor — JavaScript/JSON Mode

Replace the two `<textarea>` elements in `editAdminExtension()` (`admin-handlers.js` line 706-710) with CM6 instances.

**Behavior:**

- Manifest textarea → JSON mode with bracket matching, auto-indent
- Script textarea → JavaScript mode with bracket matching, auto-indent
- Line numbers enabled
- Search/replace (Ctrl+F) — currently impossible in a textarea
- Tab key inserts proper indent (replaces the manual keydown handler at line 726-736)
- Themed to match the app's dark/light mode

**Factory:**

```javascript
CM.codeEditor(document.getElementById('extEdit-manifest-container'), {
    language: 'json',
    value: manifestJSON,
    lineNumbers: true,
    darkMode: isDark,
});

CM.codeEditor(document.getElementById('extEdit-script-container'), {
    language: 'javascript',
    value: script,
    lineNumbers: true,
    darkMode: isDark,
    keymap: prefs.editorKeymap || 'standard',  // 'standard' | 'vim' | 'emacs'
});
```

**Save integration:** `saveAdminExtension()` currently reads `.value` from the textareas. The CM6 instances expose `.getValue()` — update the save function to call that instead.

### 3. Future Code Surface (Roadmap)

The `CM.codeEditor()` factory with language auto-detection covers this. No additional work needed now — just document that it's available for the extensions surface type when that ships.

---

## Loading Strategy

The CM6 bundle is **not** in `SHELL_FILES` (service worker precache list). It loads on demand:

```html
<!-- index.html: load after core app scripts, before app.js init -->
<script src="vendor/codemirror/codemirror.bundle.js?v=%%APP_VERSION%%"
        onerror="console.warn('CM6 not available — falling back to textarea')"></script>
<link rel="stylesheet" href="vendor/codemirror/codemirror.bundle.css?v=%%APP_VERSION%%"
      onerror="this.remove()">
```

**Graceful degradation:** If the bundle fails to load (disconnected environment without vendored copy, build issue, etc.), `window.CM` is undefined and all integration points fall back to plain `<textarea>`. Each factory callsite checks:

```javascript
if (window.CM) {
    CM.chatInput(container, opts);
} else {
    // existing textarea behavior — unchanged
}
```

This means CM6 is a progressive enhancement, not a hard dependency.

---

## SW Cache Considerations

Add `vendor/codemirror/` to the SW exclusion list (same fix as the extensions path discussed earlier):

```javascript
if (url.pathname.includes('/api/') ||
    url.pathname.includes('/ws') ||
    url.pathname.includes('/branding/') ||
    url.pathname.includes('/extensions/') ||
    url.pathname.includes('/vendor/codemirror/') ||
    event.request.method !== 'GET') {
    return;
}
```

Or, more broadly, exclude all of `/vendor/` if you want vendor updates to always bypass the SW cache. The core vendors (marked, purify) are small enough that network-first is fine.

Alternatively, since the bundle is versioned via `?v=%%APP_VERSION%%` and only changes on new builds, it could safely be in `SHELL_FILES` for precaching — unlike extensions which are user-editable at runtime. Either approach works.

---

## Theme Integration

CM6's `oneDark` theme for dark mode. For light mode, the CM6 default is clean and neutral.

Both need CSS variable overrides to match the app's existing `--bg`, `--bg-2`, `--border`, `--text`, `--accent` palette:

```javascript
const switchboardTheme = EditorView.theme({
    '&': {
        backgroundColor: 'var(--bg)',
        color: 'var(--text)',
        fontSize: '14px',
    },
    '.cm-content': {
        fontFamily: 'var(--mono)',
        caretColor: 'var(--accent)',
    },
    '.cm-cursor': {
        borderLeftColor: 'var(--accent)',
    },
    '&.cm-focused .cm-selectionBackground, .cm-selectionBackground': {
        backgroundColor: 'var(--accent-muted, rgba(99,102,241,0.2))',
    },
    '.cm-gutters': {
        backgroundColor: 'var(--bg-2)',
        color: 'var(--text-3)',
        borderRight: '1px solid var(--border)',
    },
}, { dark: document.body.classList.contains('dark-theme') });
```

This goes into the bundle so it's applied automatically.

---

## File Layout

```
src/
  editor/
    package.json          # CM6 deps + esbuild
    build.mjs             # esbuild script
    index.mjs             # bundle entrypoint, exports window.CM
    theme.mjs             # switchboard theme overrides
    chat-input.mjs        # chatInput() factory
    code-editor.mjs       # codeEditor() factory
  js/
    ...                   # existing app code (unchanged)
  vendor/
    marked.min.js         # existing
    purify.min.js         # existing
    codemirror/           # gitignored — built by Docker
      codemirror.bundle.js
      codemirror.bundle.css
```

The `src/editor/` directory is self-contained. It has its own `package.json` and `node_modules` (inside Docker only). No npm dependencies bleed into the main `src/js/` code. The build output lands in `vendor/codemirror/` which is gitignored like the other Docker-built vendors.

---

## Dockerfile Changes

```dockerfile
# Stage 1: vendor libs (existing)
FROM node:20-alpine AS vendor
# ... existing marked, purify, mermaid, katex extraction ...

# Stage 2: CM6 bundle (NEW)
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/ ./
RUN npm ci
RUN node build.mjs

# Stage 3: nginx (existing, with additions)
FROM nginx:1-alpine
# ... existing setup ...
COPY src/ /usr/share/nginx/html/
COPY --from=vendor /vendor/ /usr/share/nginx/html/vendor/
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
# ... rest unchanged ...
```

The `vendor` and `cm6-build` stages run in parallel (Docker BuildKit), so build time impact is minimal.

---

## Migration Checklist

### Phase 1: Bundle + Extension Editor
- [ ] Create `src/editor/` with package.json, build.mjs, index.mjs
- [ ] Implement `CM.codeEditor()` factory
- [ ] Update Dockerfile.frontend with cm6-build stage
- [ ] Replace extension editor textareas in `admin-handlers.js`
- [ ] Remove manual Tab key handler (CM6 handles it)
- [ ] Verify save/load round-trip with CM6 `.getValue()`
- [ ] Add graceful fallback if `window.CM` undefined

### Phase 2: Chat Input
- [ ] Implement `CM.chatInput()` factory with markdown mode
- [ ] Wire Enter=send, Shift+Enter=newline keybindings
- [ ] Integrate with `updateInputTokens()` via onChange callback
- [ ] Update `chat.js`, `attachments.js`, `tokens.js` to use CM API
- [ ] Auto-grow height behavior matching current textarea
- [ ] Test paste handling (plain text, code, attachments)
- [ ] Test mobile/touch input

### Phase 3: Polish
- [ ] Theme integration with CSS variables
- [ ] Dark/light mode switching (listen for theme toggle)
- [ ] Editor keymap preference UI (Standard / Vim / Emacs) in appearance settings
- [ ] Vim/Emacs modes on code editor + extension editor only (not chat input)
- [ ] Add to `SHELL_FILES` in sw.js (or exclude from SW cache)
- [ ] Update `debug.js` snapshot to include CM6 version
- [ ] Documentation in ARCHITECTURE.md

---

## Bundle Size Estimate

Based on CM6 package sizes (minified + tree-shaken by esbuild):

| Package | Approx Size |
|---------|-------------|
| @codemirror/view + state + commands | ~90KB |
| @codemirror/language + highlight | ~30KB |
| lang-markdown + lang-javascript + lang-json | ~40KB |
| lang-go + lang-sql + lang-html + lang-css + lang-yaml | ~50KB |
| lang-python + lang-rust | ~20KB |
| theme-one-dark | ~5KB |
| search + autocomplete | ~20KB |
| @replit/codemirror-vim + emacs | ~40KB |
| **Total (minified)** | **~295KB** |
| **Gzipped** | **~90KB** |

For comparison: mermaid.min.js is ~1.2MB. This is modest.

---

## Resolved Decisions

1. **Language modes** — Ship Python, Rust, and YAML upfront alongside Go, JS, JSON, SQL, HTML, CSS, and Markdown. Covers the team's stack. Additional modes are a one-line import + rebuild.

2. **Local dev build script** — Provide a `scripts/build-editor.sh` that runs the esbuild step standalone, shared by both Dockerfile.frontend and the unified Dockerfile. Developers can also run it locally to get the CM6 bundle without a full Docker build.

3. **Vim/Emacs keybindings** — Ship both. Exposed as a user preference toggle (default: standard). Applied via `CM.keybindings.vim` / `CM.keybindings.emacs` as CM6 extensions at editor creation time.

---

## Build Script

Shared script used by both Dockerfiles and local dev:

```bash
#!/bin/sh
# scripts/build-editor.sh — Build CM6 bundle
# Usage: ./scripts/build-editor.sh [output-dir]
#
# Defaults to src/vendor/codemirror/ for local dev.
# Dockerfiles pass /build/dist/ as the output dir.
set -e

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
EDITOR_DIR="${SCRIPT_DIR}/../src/editor"
OUTPUT_DIR="${1:-${SCRIPT_DIR}/../src/vendor/codemirror}"

cd "${EDITOR_DIR}"

# Install if needed (CI/Docker will already have node_modules)
[ -d node_modules ] || npm ci

mkdir -p "${OUTPUT_DIR}"
node build.mjs --outdir="${OUTPUT_DIR}"

echo "✅ CM6 bundle → ${OUTPUT_DIR}"
```

Dockerfile usage:

```dockerfile
FROM node:20-alpine AS cm6-build
WORKDIR /build
COPY src/editor/ /build/src/editor/
COPY scripts/build-editor.sh /build/scripts/build-editor.sh
RUN cd /build/src/editor && npm ci
RUN sh /build/scripts/build-editor.sh /build/dist
```

---

## Keybinding Preference

Add to user appearance preferences (alongside dark/light theme toggle):

```
Editor keybindings: [Standard ▾] [Vim] [Emacs]
```

Stored in `localStorage` under `cs-appearance`:

```javascript
const prefs = JSON.parse(localStorage.getItem('cs-appearance') || '{}');
// prefs.editorKeymap = 'standard' | 'vim' | 'emacs'
```

Applied at editor creation time:

```javascript
const keymapExt = [];
if (prefs.editorKeymap === 'vim') keymapExt.push(CM.keybindings.vim());
else if (prefs.editorKeymap === 'emacs') keymapExt.push(CM.keybindings.emacs());

CM.codeEditor(target, {
    language: 'javascript',
    extensions: keymapExt,
    // ...
});
```

**Note:** Vim/Emacs modes apply only to the code editor and extension editor surfaces. The chat input always uses standard keybindings to avoid confusing Enter-to-send behavior with modal editing.