core/docs/DESIGN-surface-runners.md

# DESIGN: Surface Runners — v0.7.1–v0.7.3

## Status: Proposed

## Problem

Armature has two test tiers today:

1. **Go unit tests** — test store methods, handlers, middleware, sandbox.
   Run in CI on every push. Coverage is good for kernel internals.

2. **ICD/SDK test runners** — browser-based test suites that validate API
   endpoint contracts and SDK domain methods. Run manually by navigating
   to `/s/icd-test-runner` or `/s/sdk-test-runner`.

Neither tier catches the class of bugs discovered during manual testing:

- **Cross-surface state:** Notification bell state not syncing, announcement
  dismiss not persisting across surface navigations.
- **Package integration:** Workflow demo shows "not installed" because its
  API call fails silently. The API works (unit tests pass); the surface's
  integration with the API is broken.
- **Test side-effects:** ICD security tests install `evil.surface` with no
  cleanup — it leaks into the menu.
- **Surface lifecycle:** Surfaces fail to load, mount into wrong containers,
  miss SDK boot, or render without shell chrome.

These are integration bugs — they live at the boundary between kernel and
package, between surface and surface, between API and UI. They require a
new test tier.

## Solution Overview

Three deliverables across three versions:

| Version | Deliverable | What it catches |
|---------|-------------|-----------------|
| v0.7.1 | Runner framework (`sw.testing`) | Framework bugs, standardizes existing runners |
| v0.7.2 | Package runners + CI gate | Package integration bugs, regressions |
| v0.7.3 | Headless E2E (Playwright) | DOM rendering bugs, navigation flows, visual regressions |

## v0.7.1 — Runner Framework

### `sw.testing` SDK Module

New kernel SDK module at `src/js/sw/sdk/testing.js`. Provides structured
test authoring, lifecycle hooks, cleanup tracking, and machine-readable
results.

```js
// Extension runner registers suites during load
sw.testing.suite('notes-crud', async (s) => {
    let folderId, noteId;

    s.beforeAll(async () => {
        // Setup: create a test folder
        const r = await sw.api.post('/api/v1/ext/notes/folders', {
            name: 'test-' + Date.now()
        });
        folderId = r.id;
        s.track('folder', folderId); // auto-cleanup
    });

    s.test('create note', async (t) => {
        const r = await sw.api.post('/api/v1/ext/notes/notes', {
            title: 'Test Note', folder_id: folderId, content: '# Hello'
        });
        t.assert.ok(r.id, 'note has ID');
        t.assert.eq(r.title, 'Test Note');
        noteId = r.id;
        t.track('note', noteId); // auto-cleanup
    });

    s.test('renderers fire', async (t) => {
        // Test that mermaid block in note content triggers renderer
        await sw.api.patch('/api/v1/ext/notes/notes/' + noteId, {
            content: '```mermaid\ngraph LR; A-->B\n```'
        });
        // Renderer integration tested via DOM assertion
        // (only meaningful in headless E2E — marked as browser-only)
        t.browserOnly(() => {
            const el = document.querySelector('.mermaid svg');
            t.assert.ok(el, 'mermaid rendered to SVG');
        });
    });

    s.afterAll(async () => {
        // s.track() resources auto-cleaned here
        // Manual cleanup for anything not tracked
    });
});
```

### Core API

```js
sw.testing.suite(name, fn)        // Register a test suite
sw.testing.run(name?)             // Run one suite or all
sw.testing.results()              // Get structured results (JSON)
sw.testing.on('complete', fn)     // Event when run finishes

// Inside suite:
s.test(name, fn)                  // Register a test
s.beforeAll(fn)                   // Runs once before all tests
s.afterAll(fn)                    // Runs once after all tests (always, even on failure)
s.beforeEach(fn)                  // Runs before each test
s.afterEach(fn)                   // Runs after each test
s.track(type, id)                 // Register resource for auto-cleanup
s.skip(reason)                    // Skip entire suite

// Inside test:
t.assert.ok(val, msg)             // Truthy
t.assert.eq(a, b, msg)           // Deep equality
t.assert.neq(a, b, msg)          // Not equal
t.assert.gt(a, b, msg)           // Greater than
t.assert.match(str, re, msg)     // Regex match
t.assert.throws(fn, msg)         // Expects throw
t.assert.status(resp, code, msg) // HTTP status check
t.track(type, id)                // Register resource for auto-cleanup
t.warn(msg)                      // Emit warning (non-fatal)
t.browserOnly(fn)                // Only runs in headless E2E, skipped in API-only mode
t.skip(reason)                   // Skip this test
```

### `requires` Declarations

Runner packages declare dependencies in their manifest:

```json
{
    "id": "chat-runner",
    "type": "test-runner",
    "title": "Chat Runner",
    "requires": ["chat", "chat-core"],
    "version": "0.1.0"
}
```

On load, the framework calls `GET /api/v1/surfaces` (or equivalent) to
check which packages are installed. If any `requires` entry is missing:

- Suite is marked `skipped` with reason: `"Missing required package: chat-core"`
- No tests execute — clean skip, not a failure
- The runner registry surface shows the skip reason prominently

This directly solves the "workflow demo shows not-installed" pattern:
the runner *knows* what should be installed and reports clearly when it isn't.

### Auto-Cleanup

The `track(type, id)` method registers resources for deletion in `afterAll`.
Supported resource types and their cleanup endpoints:

| Type | Cleanup Action |
|------|---------------|
| `channel` | `DELETE /api/v1/channels/:id` |
| `note` | `DELETE /api/v1/ext/notes/notes/:id` |
| `folder` | `DELETE /api/v1/ext/notes/folders/:id` |
| `workflow` | `DELETE /api/v1/workflows/:id` |
| `schedule` | `DELETE /api/v1/schedules/:id` |
| `package` | `DELETE /api/v1/admin/packages/:id` |
| `user` | `DELETE /api/v1/admin/users/:id` |
| `team` | `DELETE /api/v1/admin/teams/:id` |

Cleanup runs in reverse order (LIFO) in `afterAll`, regardless of
test pass/fail. Cleanup failures are reported as warnings, not failures.
The framework never swallows cleanup errors silently.

### Result Structure

```json
{
    "runner": "notes-runner",
    "timestamp": "2026-04-01T12:00:00Z",
    "duration_ms": 1234,
    "summary": { "total": 5, "passed": 4, "failed": 0, "warned": 1, "skipped": 0 },
    "suites": [
        {
            "name": "notes-crud",
            "status": "passed",
            "duration_ms": 890,
            "tests": [
                {
                    "name": "create note",
                    "status": "passed",
                    "duration_ms": 120,
                    "warnings": [],
                    "cleanup": { "tracked": 1, "cleaned": 1, "failed": 0 }
                }
            ]
        }
    ],
    "requires": { "met": ["notes"], "missing": [] }
}
```

### Warning Tier

Three result statuses:

- **`passed`** — assertions all passed, cleanup succeeded
- **`failed`** — at least one assertion failed
- **`warned`** — assertions passed but something non-fatal happened:
  - API returned unexpected shape (extra/missing fields) but test doesn't depend on the exact field
  - Cleanup failed for a tracked resource
  - Timing exceeded a soft threshold
  - `t.warn(msg)` called explicitly

Warnings are **never silent.** They appear in the UI and structured results.
The difference from the current `catch (e) { /* ignore */ }` pattern is
that warnings are *visible* — a human or CI system can decide whether
to investigate.

### ICD/SDK Runner Migration

The existing runners use a hand-rolled framework (`T.test()`, `T.assert()`,
`T.authFetch()`). Migration preserves all test logic:

| Current | New |
|---------|-----|
| `T.test(tier, group, name, fn)` | `s.test(name, fn)` inside `sw.testing.suite(tier + '/' + group, fn)` |
| `T.assert(cond, msg)` | `t.assert.ok(cond, msg)` |
| `T.authFetch(token, method, path, body)` | Kept as utility — not an assertion primitive |
| `T.apiPost(...)` | Kept as utility |
| Result rendering (`ui.js`) | Delegated to runner registry surface |
| No cleanup hooks | `s.track()` + `s.afterAll()` |

The ICD and SDK runners become packages with `"type": "test-runner"` in
their manifests. Their existing surfaces (`/s/icd-test-runner`,
`/s/sdk-test-runner`) are replaced by the unified runner registry at
`/s/test-runners`.

## v0.7.2 — Package Runners

### Runner Inventory

| Runner | `requires` | Key Assertions |
|--------|-----------|---------------|
| `notes-runner` | `["notes"]` | CRUD, folders, tags, backlinks, search, markdown rendering, SDK integration |
| `chat-runner` | `["chat", "chat-core"]` | Channel CRUD, messaging, participant display, renderer blocks in messages |
| `schedules-runner` | `["schedules"]` | Schedule CRUD, cron expression, toggle, Starlark exec |
| `workflow-runner` | `["content-approval"]` | Install detection, stage progression, form submission, signoff |
| `renderer-runner` | `["mermaid-renderer"]` | `sw.renderers.register` contract, post-render hooks, block rendering |

### Runner Result API

New kernel endpoints (no package required — kernel-provided):

```
POST /api/v1/test-runners/run          → Run all installed runners
POST /api/v1/test-runners/run/:id      → Run specific runner
GET  /api/v1/test-runners/results      → Last run results (JSON)
GET  /api/v1/test-runners/results/:id  → Last run results for specific runner
```

These endpoints enable CI to trigger and consume runner results via
`curl` without browser automation. The v0.7.3 Playwright harness is
additive — not required for CI gating.

**Auth:** Admin-only. Runners create/delete resources — they must run
with elevated permissions.

### CI Integration

New stage in `.gitea/workflows/ci.yaml`:

```yaml
test-runners:
    needs: [unit-tests]
    steps:
        - name: Boot server
          run: |
              docker compose up -d
              ./ci/wait-for-healthy.sh
        - name: Run surface runners
          run: |
              RESULT=$(curl -s -X POST http://localhost:8080/api/v1/test-runners/run \
                  -H "Authorization: Bearer $ADMIN_TOKEN")
              FAILED=$(echo "$RESULT" | jq '.summary.failed')
              if [ "$FAILED" != "0" ]; then
                  echo "$RESULT" | jq '.suites[] | select(.status == "failed")'
                  exit 1
              fi
```

Runs in both PG and SQLite pipelines. Server boots with `BUNDLED_PACKAGES=*`
so all packages and their runners are installed.

## v0.7.3 — Headless E2E

### Playwright Harness

`ci/e2e-surface-test.sh`:

1. `docker compose up -d` (server + DB)
2. `npx playwright install chromium` (CI caches this)
3. Run `ci/e2e-surfaces.spec.ts`
4. Collect screenshots on failure
5. `docker compose down`

### Surface Navigation Smoke Test

```ts
test('all surfaces reachable', async ({ page }) => {
    // Login
    await page.goto('/');
    await page.fill('#username', 'admin');
    await page.fill('#password', 'admin');
    await page.click('button[type="submit"]');

    // Navigate through every installed surface
    const surfaces = ['notes', 'chat', 'admin', 'settings', 'docs'];
    for (const s of surfaces) {
        await page.goto(`/s/${s}`);
        // Assert: page loaded, no uncaught JS errors
        await expect(page.locator('.sw-topbar')).toBeVisible();
        // Assert: home link works
        await page.click('.sw-topbar__home');
        await expect(page).toHaveURL('/');
    }
});
```

This is the automated version of "hello dashboard has no way out" — if
any surface fails to render a topbar or its home link doesn't work, CI
catches it.

### Screenshot on Failure

```ts
test.afterEach(async ({ page }, testInfo) => {
    if (testInfo.status !== 'passed') {
        await page.screenshot({
            path: `ci/artifacts/failure-${testInfo.title}.png`,
            fullPage: true
        });
        const logs = await page.evaluate(() =>
            (window.__consoleErrors || []).join('\n')
        );
        fs.writeFileSync(
            `ci/artifacts/console-${testInfo.title}.log`, logs
        );
    }
});
```

Artifacts saved to CI workspace. On failure, the developer gets a
screenshot + console log dump without needing to reproduce locally.

### Visual Regression (Optional)

Not a CI gate in v0.7.3 — produces a diff report for human review:

```ts
test('visual baseline - notes', async ({ page }) => {
    await page.goto('/s/notes');
    await expect(page.locator('.sw-topbar')).toBeVisible();
    await expect(page).toHaveScreenshot('notes.png', {
        maxDiffPixelRatio: 0.01
    });
});
```

Playwright stores baseline screenshots in `ci/visual-baselines/`.
`toHaveScreenshot` compares against baseline and produces a diff image
on mismatch. Foundation for future visual regression gating.

## Sequencing

```
v0.7.0  Shell Contract        ← prerequisite: surfaces need topbar before
  │                              runners can assert on it
  ↓
v0.7.1  Runner Framework      ← standardize test authoring
  │
  ↓
v0.7.2  Package Runners + CI  ← write the actual tests, wire into CI
  │
  ↓
v0.7.3  Headless E2E          ← automate browser-based runner execution
```

Each version is independently shippable. v0.7.2's API-based CI gate
works without v0.7.3's Playwright. v0.7.3 adds coverage for DOM-level
bugs that API-only runners can't catch.

## Open Questions

1. **Runner package type.** Should `"type": "test-runner"` be a new
   manifest type, or should runners be `"type": "surface"` with a
   `"tags": ["test-runner"]` convention? New type is cleaner but requires
   a `ValidateManifest()` update.

2. **Runner discovery.** The registry surface needs to find all installed
   runners. Options: (a) scan installed packages for `type: "test-runner"`,
   (b) runners register themselves via `sw.testing.register()` during SDK
   boot. Option (a) is declarative and doesn't require runner JS to load
   before discovery.

3. **Parallel vs sequential.** Should runners execute in parallel?
   Probably not initially — shared DB state means test isolation is hard.
   Sequential is safer. Parallel can be a future optimization.

4. **SQLite limitations.** Some runners (cluster, multi-node) are
   PG-only. The `requires` mechanism should support
   `"requires_db": "postgres"` for these cases, or runners should
   self-skip when `sw.config.db_driver === 'sqlite'`.