core/docs/DESIGN-surface-runners.md

DESIGN: Surface Runners — v0.7.1–v0.7.3

Status: Proposed

Problem

Armature has two test tiers today:

1. Go unit tests — test store methods, handlers, middleware, sandbox. Run in CI on every push. Coverage is good for kernel internals.

2. ICD/SDK test runners — browser-based test suites that validate API endpoint contracts and SDK domain methods. Run manually by navigating to /s/icd-test-runner or /s/sdk-test-runner.

Neither tier catches the class of bugs discovered during manual testing:

• Cross-surface state: Notification bell state not syncing, announcement dismiss not persisting across surface navigations.
• Package integration: Workflow demo shows "not installed" because its API call fails silently. The API works (unit tests pass); the surface's integration with the API is broken.
• Test side-effects: ICD security tests install evil.surface with no cleanup — it leaks into the menu.
• Surface lifecycle: Surfaces fail to load, mount into wrong containers, miss SDK boot, or render without shell chrome.

These are integration bugs — they live at the boundary between kernel and package, between surface and surface, between API and UI. They require a new test tier.

Solution Overview

Three deliverables across three versions:

Version	Deliverable	What it catches
v0.7.1	Runner framework (sw.testing)	Framework bugs, standardizes existing runners
v0.7.2	Package runners + CI gate	Package integration bugs, regressions
v0.7.3	Headless E2E (Playwright)	DOM rendering bugs, navigation flows, visual regressions

v0.7.1 — Runner Framework

sw.testing SDK Module

New kernel SDK module at src/js/sw/sdk/testing.js. Provides structured test authoring, lifecycle hooks, cleanup tracking, and machine-readable results.

// Extension runner registers suites during load
sw.testing.suite('notes-crud', async (s) => {
    let folderId, noteId;

    s.beforeAll(async () => {
        // Setup: create a test folder
        const r = await sw.api.post('/api/v1/ext/notes/folders', {
            name: 'test-' + Date.now()
        });
        folderId = r.id;
        s.track('folder', folderId); // auto-cleanup
    });

    s.test('create note', async (t) => {
        const r = await sw.api.post('/api/v1/ext/notes/notes', {
            title: 'Test Note', folder_id: folderId, content: '# Hello'
        });
        t.assert.ok(r.id, 'note has ID');
        t.assert.eq(r.title, 'Test Note');
        noteId = r.id;
        t.track('note', noteId); // auto-cleanup
    });

    s.test('renderers fire', async (t) => {
        // Test that mermaid block in note content triggers renderer
        await sw.api.patch('/api/v1/ext/notes/notes/' + noteId, {
            content: '```mermaid\ngraph LR; A-->B\n```'
        });
        // Renderer integration tested via DOM assertion
        // (only meaningful in headless E2E — marked as browser-only)
        t.browserOnly(() => {
            const el = document.querySelector('.mermaid svg');
            t.assert.ok(el, 'mermaid rendered to SVG');
        });
    });

    s.afterAll(async () => {
        // s.track() resources auto-cleaned here
        // Manual cleanup for anything not tracked
    });
});

Core API

sw.testing.suite(name, fn)        // Register a test suite
sw.testing.run(name?)             // Run one suite or all
sw.testing.results()              // Get structured results (JSON)
sw.testing.on('complete', fn)     // Event when run finishes

// Inside suite:
s.test(name, fn)                  // Register a test
s.beforeAll(fn)                   // Runs once before all tests
s.afterAll(fn)                    // Runs once after all tests (always, even on failure)
s.beforeEach(fn)                  // Runs before each test
s.afterEach(fn)                   // Runs after each test
s.track(type, id)                 // Register resource for auto-cleanup
s.skip(reason)                    // Skip entire suite

// Inside test:
t.assert.ok(val, msg)             // Truthy
t.assert.eq(a, b, msg)           // Deep equality
t.assert.neq(a, b, msg)          // Not equal
t.assert.gt(a, b, msg)           // Greater than
t.assert.match(str, re, msg)     // Regex match
t.assert.throws(fn, msg)         // Expects throw
t.assert.status(resp, code, msg) // HTTP status check
t.track(type, id)                // Register resource for auto-cleanup
t.warn(msg)                      // Emit warning (non-fatal)
t.browserOnly(fn)                // Only runs in headless E2E, skipped in API-only mode
t.skip(reason)                   // Skip this test

requires Declarations

Runner packages declare dependencies in their manifest:

{
    "id": "chat-runner",
    "type": "test-runner",
    "title": "Chat Runner",
    "requires": ["chat", "chat-core"],
    "version": "0.1.0"
}

On load, the framework calls GET /api/v1/surfaces (or equivalent) to check which packages are installed. If any requires entry is missing:

• Suite is marked skipped with reason: "Missing required package: chat-core"
• No tests execute — clean skip, not a failure
• The runner registry surface shows the skip reason prominently

This directly solves the "workflow demo shows not-installed" pattern: the runner knows what should be installed and reports clearly when it isn't.

Auto-Cleanup

The track(type, id) method registers resources for deletion in afterAll. Supported resource types and their cleanup endpoints:

Type	Cleanup Action
channel	DELETE /api/v1/channels/:id
note	DELETE /api/v1/ext/notes/notes/:id
folder	DELETE /api/v1/ext/notes/folders/:id
workflow	DELETE /api/v1/workflows/:id
schedule	DELETE /api/v1/schedules/:id
package	DELETE /api/v1/admin/packages/:id
user	DELETE /api/v1/admin/users/:id
team	DELETE /api/v1/admin/teams/:id

Cleanup runs in reverse order (LIFO) in afterAll, regardless of test pass/fail. Cleanup failures are reported as warnings, not failures. The framework never swallows cleanup errors silently.

Result Structure

{
    "runner": "notes-runner",
    "timestamp": "2026-04-01T12:00:00Z",
    "duration_ms": 1234,
    "summary": { "total": 5, "passed": 4, "failed": 0, "warned": 1, "skipped": 0 },
    "suites": [
        {
            "name": "notes-crud",
            "status": "passed",
            "duration_ms": 890,
            "tests": [
                {
                    "name": "create note",
                    "status": "passed",
                    "duration_ms": 120,
                    "warnings": [],
                    "cleanup": { "tracked": 1, "cleaned": 1, "failed": 0 }
                }
            ]
        }
    ],
    "requires": { "met": ["notes"], "missing": [] }
}

Warning Tier

Three result statuses:

• passed — assertions all passed, cleanup succeeded
• failed — at least one assertion failed
• warned — assertions passed but something non-fatal happened:
  • API returned unexpected shape (extra/missing fields) but test doesn't depend on the exact field
  • Cleanup failed for a tracked resource
  • Timing exceeded a soft threshold
  • t.warn(msg) called explicitly

Warnings are never silent. They appear in the UI and structured results. The difference from the current catch (e) { /* ignore */ } pattern is that warnings are visible — a human or CI system can decide whether to investigate.

ICD/SDK Runner Migration

The existing runners use a hand-rolled framework (T.test(), T.assert(), T.authFetch()). Migration preserves all test logic:

Current	New
T.test(tier, group, name, fn)	s.test(name, fn) inside sw.testing.suite(tier + '/' + group, fn)
T.assert(cond, msg)	t.assert.ok(cond, msg)
T.authFetch(token, method, path, body)	Kept as utility — not an assertion primitive
T.apiPost(...)	Kept as utility
Result rendering (ui.js)	Delegated to runner registry surface
No cleanup hooks	s.track() + s.afterAll()

The ICD and SDK runners become packages with "type": "test-runner" in their manifests. Their existing surfaces (/s/icd-test-runner, /s/sdk-test-runner) are replaced by the unified runner registry at /s/test-runners.

v0.7.2 — Package Runners

Runner Inventory

Runner	requires	Key Assertions
notes-runner	["notes"]	CRUD, folders, tags, backlinks, search, markdown rendering, SDK integration
chat-runner	["chat", "chat-core"]	Channel CRUD, messaging, participant display, renderer blocks in messages
schedules-runner	["schedules"]	Schedule CRUD, cron expression, toggle, Starlark exec
workflow-runner	["content-approval"]	Install detection, stage progression, form submission, signoff
renderer-runner	["mermaid-renderer"]	sw.renderers.register contract, post-render hooks, block rendering

Runner Result API

New kernel endpoints (no package required — kernel-provided):

POST /api/v1/test-runners/run          → Run all installed runners
POST /api/v1/test-runners/run/:id      → Run specific runner
GET  /api/v1/test-runners/results      → Last run results (JSON)
GET  /api/v1/test-runners/results/:id  → Last run results for specific runner

These endpoints enable CI to trigger and consume runner results via curl without browser automation. The v0.7.3 Playwright harness is additive — not required for CI gating.

Auth: Admin-only. Runners create/delete resources — they must run with elevated permissions.

CI Integration

New stage in .gitea/workflows/ci.yaml:

test-runners:
    needs: [unit-tests]
    steps:
        - name: Boot server
          run: |
              docker compose up -d
              ./ci/wait-for-healthy.sh
        - name: Run surface runners
          run: |
              RESULT=$(curl -s -X POST http://localhost:8080/api/v1/test-runners/run \
                  -H "Authorization: Bearer $ADMIN_TOKEN")
              FAILED=$(echo "$RESULT" | jq '.summary.failed')
              if [ "$FAILED" != "0" ]; then
                  echo "$RESULT" | jq '.suites[] | select(.status == "failed")'
                  exit 1
              fi

Runs in both PG and SQLite pipelines. Server boots with BUNDLED_PACKAGES=* so all packages and their runners are installed.

v0.7.3 — Headless E2E

Playwright Harness

ci/e2e-surface-test.sh:

1. docker compose up -d (server + DB)
2. npx playwright install chromium (CI caches this)
3. Run ci/e2e-surfaces.spec.ts
4. Collect screenshots on failure
5. docker compose down

Surface Navigation Smoke Test

test('all surfaces reachable', async ({ page }) => {
    // Login
    await page.goto('/');
    await page.fill('#username', 'admin');
    await page.fill('#password', 'admin');
    await page.click('button[type="submit"]');

    // Navigate through every installed surface
    const surfaces = ['notes', 'chat', 'admin', 'settings', 'docs'];
    for (const s of surfaces) {
        await page.goto(`/s/${s}`);
        // Assert: page loaded, no uncaught JS errors
        await expect(page.locator('.sw-topbar')).toBeVisible();
        // Assert: home link works
        await page.click('.sw-topbar__home');
        await expect(page).toHaveURL('/');
    }
});

This is the automated version of "hello dashboard has no way out" — if any surface fails to render a topbar or its home link doesn't work, CI catches it.

Screenshot on Failure

test.afterEach(async ({ page }, testInfo) => {
    if (testInfo.status !== 'passed') {
        await page.screenshot({
            path: `ci/artifacts/failure-${testInfo.title}.png`,
            fullPage: true
        });
        const logs = await page.evaluate(() =>
            (window.__consoleErrors || []).join('\n')
        );
        fs.writeFileSync(
            `ci/artifacts/console-${testInfo.title}.log`, logs
        );
    }
});

Artifacts saved to CI workspace. On failure, the developer gets a screenshot + console log dump without needing to reproduce locally.

Visual Regression (Optional)

Not a CI gate in v0.7.3 — produces a diff report for human review:

test('visual baseline - notes', async ({ page }) => {
    await page.goto('/s/notes');
    await expect(page.locator('.sw-topbar')).toBeVisible();
    await expect(page).toHaveScreenshot('notes.png', {
        maxDiffPixelRatio: 0.01
    });
});

Playwright stores baseline screenshots in ci/visual-baselines/. toHaveScreenshot compares against baseline and produces a diff image on mismatch. Foundation for future visual regression gating.

Sequencing

v0.7.0  Shell Contract        ← prerequisite: surfaces need topbar before
  │                              runners can assert on it
  ↓
v0.7.1  Runner Framework      ← standardize test authoring
  │
  ↓
v0.7.2  Package Runners + CI  ← write the actual tests, wire into CI
  │
  ↓
v0.7.3  Headless E2E          ← automate browser-based runner execution

Each version is independently shippable. v0.7.2's API-based CI gate works without v0.7.3's Playwright. v0.7.3 adds coverage for DOM-level bugs that API-only runners can't catch.

Open Questions

1. Runner package type. Should "type": "test-runner" be a new manifest type, or should runners be "type": "surface" with a "tags": ["test-runner"] convention? New type is cleaner but requires a ValidateManifest() update.

2. Runner discovery. The registry surface needs to find all installed runners. Options: (a) scan installed packages for type: "test-runner", (b) runners register themselves via sw.testing.register() during SDK boot. Option (a) is declarative and doesn't require runner JS to load before discovery.

3. Parallel vs sequential. Should runners execute in parallel? Probably not initially — shared DB state means test isolation is hard. Sequential is safer. Parallel can be a future optimization.

4. SQLite limitations. Some runners (cluster, multi-node) are PG-only. The requires mechanism should support "requires_db": "postgres" for these cases, or runners should self-skip when sw.config.db_driver === 'sqlite'.