V0.7.0 shell contract (#54)

2026-04-01 20:19:45 +00:00
parent 1236220302
commit e916ed41ea
151 changed files with 3250 additions and 684 deletions
--- a/docs/DESIGN-surface-runners.md
+++ b/docs/DESIGN-surface-runners.md
@@ -0,0 +1,407 @@
+# DESIGN: Surface Runners — v0.7.1–v0.7.3
+
+## Status: Proposed
+
+## Problem
+
+Armature has two test tiers today:
+
+1. **Go unit tests** — test store methods, handlers, middleware, sandbox.
+   Run in CI on every push. Coverage is good for kernel internals.
+
+2. **ICD/SDK test runners** — browser-based test suites that validate API
+   endpoint contracts and SDK domain methods. Run manually by navigating
+   to `/s/icd-test-runner` or `/s/sdk-test-runner`.
+
+Neither tier catches the class of bugs discovered during manual testing:
+
+- **Cross-surface state:** Notification bell state not syncing, announcement
+  dismiss not persisting across surface navigations.
+- **Package integration:** Workflow demo shows "not installed" because its
+  API call fails silently. The API works (unit tests pass); the surface's
+  integration with the API is broken.
+- **Test side-effects:** ICD security tests install `evil.surface` with no
+  cleanup — it leaks into the menu.
+- **Surface lifecycle:** Surfaces fail to load, mount into wrong containers,
+  miss SDK boot, or render without shell chrome.
+
+These are integration bugs — they live at the boundary between kernel and
+package, between surface and surface, between API and UI. They require a
+new test tier.
+
+## Solution Overview
+
+Three deliverables across three versions:
+
+| Version | Deliverable | What it catches |
+|---------|-------------|-----------------|
+| v0.7.1 | Runner framework (`sw.testing`) | Framework bugs, standardizes existing runners |
+| v0.7.2 | Package runners + CI gate | Package integration bugs, regressions |
+| v0.7.3 | Headless E2E (Playwright) | DOM rendering bugs, navigation flows, visual regressions |
+
+## v0.7.1 — Runner Framework
+
+### `sw.testing` SDK Module
+
+New kernel SDK module at `src/js/sw/sdk/testing.js`. Provides structured
+test authoring, lifecycle hooks, cleanup tracking, and machine-readable
+results.
+
+```js
+// Extension runner registers suites during load
+sw.testing.suite('notes-crud', async (s) => {
+    let folderId, noteId;
+
+    s.beforeAll(async () => {
+        // Setup: create a test folder
+        const r = await sw.api.post('/api/v1/ext/notes/folders', {
+            name: 'test-' + Date.now()
+        });
+        folderId = r.id;
+        s.track('folder', folderId); // auto-cleanup
+    });
+
+    s.test('create note', async (t) => {
+        const r = await sw.api.post('/api/v1/ext/notes/notes', {
+            title: 'Test Note', folder_id: folderId, content: '# Hello'
+        });
+        t.assert.ok(r.id, 'note has ID');
+        t.assert.eq(r.title, 'Test Note');
+        noteId = r.id;
+        t.track('note', noteId); // auto-cleanup
+    });
+
+    s.test('renderers fire', async (t) => {
+        // Test that mermaid block in note content triggers renderer
+        await sw.api.patch('/api/v1/ext/notes/notes/' + noteId, {
+            content: '```mermaid\ngraph LR; A-->B\n```'
+        });
+        // Renderer integration tested via DOM assertion
+        // (only meaningful in headless E2E — marked as browser-only)
+        t.browserOnly(() => {
+            const el = document.querySelector('.mermaid svg');
+            t.assert.ok(el, 'mermaid rendered to SVG');
+        });
+    });
+
+    s.afterAll(async () => {
+        // s.track() resources auto-cleaned here
+        // Manual cleanup for anything not tracked
+    });
+});
+```
+
+### Core API
+
+```js
+sw.testing.suite(name, fn)        // Register a test suite
+sw.testing.run(name?)             // Run one suite or all
+sw.testing.results()              // Get structured results (JSON)
+sw.testing.on('complete', fn)     // Event when run finishes
+
+// Inside suite:
+s.test(name, fn)                  // Register a test
+s.beforeAll(fn)                   // Runs once before all tests
+s.afterAll(fn)                    // Runs once after all tests (always, even on failure)
+s.beforeEach(fn)                  // Runs before each test
+s.afterEach(fn)                   // Runs after each test
+s.track(type, id)                 // Register resource for auto-cleanup
+s.skip(reason)                    // Skip entire suite
+
+// Inside test:
+t.assert.ok(val, msg)             // Truthy
+t.assert.eq(a, b, msg)           // Deep equality
+t.assert.neq(a, b, msg)          // Not equal
+t.assert.gt(a, b, msg)           // Greater than
+t.assert.match(str, re, msg)     // Regex match
+t.assert.throws(fn, msg)         // Expects throw
+t.assert.status(resp, code, msg) // HTTP status check
+t.track(type, id)                // Register resource for auto-cleanup
+t.warn(msg)                      // Emit warning (non-fatal)
+t.browserOnly(fn)                // Only runs in headless E2E, skipped in API-only mode
+t.skip(reason)                   // Skip this test
+```
+
+### `requires` Declarations
+
+Runner packages declare dependencies in their manifest:
+
+```json
+{
+    "id": "chat-runner",
+    "type": "test-runner",
+    "title": "Chat Runner",
+    "requires": ["chat", "chat-core"],
+    "version": "0.1.0"
+}
+```
+
+On load, the framework calls `GET /api/v1/surfaces` (or equivalent) to
+check which packages are installed. If any `requires` entry is missing:
+
+- Suite is marked `skipped` with reason: `"Missing required package: chat-core"`
+- No tests execute — clean skip, not a failure
+- The runner registry surface shows the skip reason prominently
+
+This directly solves the "workflow demo shows not-installed" pattern:
+the runner *knows* what should be installed and reports clearly when it isn't.
+
+### Auto-Cleanup
+
+The `track(type, id)` method registers resources for deletion in `afterAll`.
+Supported resource types and their cleanup endpoints:
+
+| Type | Cleanup Action |
+|------|---------------|
+| `channel` | `DELETE /api/v1/channels/:id` |
+| `note` | `DELETE /api/v1/ext/notes/notes/:id` |
+| `folder` | `DELETE /api/v1/ext/notes/folders/:id` |
+| `workflow` | `DELETE /api/v1/workflows/:id` |
+| `schedule` | `DELETE /api/v1/schedules/:id` |
+| `package` | `DELETE /api/v1/admin/packages/:id` |
+| `user` | `DELETE /api/v1/admin/users/:id` |
+| `team` | `DELETE /api/v1/admin/teams/:id` |
+
+Cleanup runs in reverse order (LIFO) in `afterAll`, regardless of
+test pass/fail. Cleanup failures are reported as warnings, not failures.
+The framework never swallows cleanup errors silently.
+
+### Result Structure
+
+```json
+{
+    "runner": "notes-runner",
+    "timestamp": "2026-04-01T12:00:00Z",
+    "duration_ms": 1234,
+    "summary": { "total": 5, "passed": 4, "failed": 0, "warned": 1, "skipped": 0 },
+    "suites": [
+        {
+            "name": "notes-crud",
+            "status": "passed",
+            "duration_ms": 890,
+            "tests": [
+                {
+                    "name": "create note",
+                    "status": "passed",
+                    "duration_ms": 120,
+                    "warnings": [],
+                    "cleanup": { "tracked": 1, "cleaned": 1, "failed": 0 }
+                }
+            ]
+        }
+    ],
+    "requires": { "met": ["notes"], "missing": [] }
+}
+```
+
+### Warning Tier
+
+Three result statuses:
+
+- **`passed`** — assertions all passed, cleanup succeeded
+- **`failed`** — at least one assertion failed
+- **`warned`** — assertions passed but something non-fatal happened:
+  - API returned unexpected shape (extra/missing fields) but test doesn't depend on the exact field
+  - Cleanup failed for a tracked resource
+  - Timing exceeded a soft threshold
+  - `t.warn(msg)` called explicitly
+
+Warnings are **never silent.** They appear in the UI and structured results.
+The difference from the current `catch (e) { /* ignore */ }` pattern is
+that warnings are *visible* — a human or CI system can decide whether
+to investigate.
+
+### ICD/SDK Runner Migration
+
+The existing runners use a hand-rolled framework (`T.test()`, `T.assert()`,
+`T.authFetch()`). Migration preserves all test logic:
+
+| Current | New |
+|---------|-----|
+| `T.test(tier, group, name, fn)` | `s.test(name, fn)` inside `sw.testing.suite(tier + '/' + group, fn)` |
+| `T.assert(cond, msg)` | `t.assert.ok(cond, msg)` |
+| `T.authFetch(token, method, path, body)` | Kept as utility — not an assertion primitive |
+| `T.apiPost(...)` | Kept as utility |
+| Result rendering (`ui.js`) | Delegated to runner registry surface |
+| No cleanup hooks | `s.track()` + `s.afterAll()` |
+
+The ICD and SDK runners become packages with `"type": "test-runner"` in
+their manifests. Their existing surfaces (`/s/icd-test-runner`,
+`/s/sdk-test-runner`) are replaced by the unified runner registry at
+`/s/test-runners`.
+
+## v0.7.2 — Package Runners
+
+### Runner Inventory
+
+| Runner | `requires` | Key Assertions |
+|--------|-----------|---------------|
+| `notes-runner` | `["notes"]` | CRUD, folders, tags, backlinks, search, markdown rendering, SDK integration |
+| `chat-runner` | `["chat", "chat-core"]` | Channel CRUD, messaging, participant display, renderer blocks in messages |
+| `schedules-runner` | `["schedules"]` | Schedule CRUD, cron expression, toggle, Starlark exec |
+| `workflow-runner` | `["content-approval"]` | Install detection, stage progression, form submission, signoff |
+| `renderer-runner` | `["mermaid-renderer"]` | `sw.renderers.register` contract, post-render hooks, block rendering |
+
+### Runner Result API
+
+New kernel endpoints (no package required — kernel-provided):
+
+```
+POST /api/v1/test-runners/run          → Run all installed runners
+POST /api/v1/test-runners/run/:id      → Run specific runner
+GET  /api/v1/test-runners/results      → Last run results (JSON)
+GET  /api/v1/test-runners/results/:id  → Last run results for specific runner
+```
+
+These endpoints enable CI to trigger and consume runner results via
+`curl` without browser automation. The v0.7.3 Playwright harness is
+additive — not required for CI gating.
+
+**Auth:** Admin-only. Runners create/delete resources — they must run
+with elevated permissions.
+
+### CI Integration
+
+New stage in `.gitea/workflows/ci.yaml`:
+
+```yaml
+test-runners:
+    needs: [unit-tests]
+    steps:
+        - name: Boot server
+          run: |
+              docker compose up -d
+              ./ci/wait-for-healthy.sh
+        - name: Run surface runners
+          run: |
+              RESULT=$(curl -s -X POST http://localhost:8080/api/v1/test-runners/run \
+                  -H "Authorization: Bearer $ADMIN_TOKEN")
+              FAILED=$(echo "$RESULT" | jq '.summary.failed')
+              if [ "$FAILED" != "0" ]; then
+                  echo "$RESULT" | jq '.suites[] | select(.status == "failed")'
+                  exit 1
+              fi
+```
+
+Runs in both PG and SQLite pipelines. Server boots with `BUNDLED_PACKAGES=*`
+so all packages and their runners are installed.
+
+## v0.7.3 — Headless E2E
+
+### Playwright Harness
+
+`ci/e2e-surface-test.sh`:
+
+1. `docker compose up -d` (server + DB)
+2. `npx playwright install chromium` (CI caches this)
+3. Run `ci/e2e-surfaces.spec.ts`
+4. Collect screenshots on failure
+5. `docker compose down`
+
+### Surface Navigation Smoke Test
+
+```ts
+test('all surfaces reachable', async ({ page }) => {
+    // Login
+    await page.goto('/');
+    await page.fill('#username', 'admin');
+    await page.fill('#password', 'admin');
+    await page.click('button[type="submit"]');
+
+    // Navigate through every installed surface
+    const surfaces = ['notes', 'chat', 'admin', 'settings', 'docs'];
+    for (const s of surfaces) {
+        await page.goto(`/s/${s}`);
+        // Assert: page loaded, no uncaught JS errors
+        await expect(page.locator('.sw-topbar')).toBeVisible();
+        // Assert: home link works
+        await page.click('.sw-topbar__home');
+        await expect(page).toHaveURL('/');
+    }
+});
+```
+
+This is the automated version of "hello dashboard has no way out" — if
+any surface fails to render a topbar or its home link doesn't work, CI
+catches it.
+
+### Screenshot on Failure
+
+```ts
+test.afterEach(async ({ page }, testInfo) => {
+    if (testInfo.status !== 'passed') {
+        await page.screenshot({
+            path: `ci/artifacts/failure-${testInfo.title}.png`,
+            fullPage: true
+        });
+        const logs = await page.evaluate(() =>
+            (window.__consoleErrors || []).join('\n')
+        );
+        fs.writeFileSync(
+            `ci/artifacts/console-${testInfo.title}.log`, logs
+        );
+    }
+});
+```
+
+Artifacts saved to CI workspace. On failure, the developer gets a
+screenshot + console log dump without needing to reproduce locally.
+
+### Visual Regression (Optional)
+
+Not a CI gate in v0.7.3 — produces a diff report for human review:
+
+```ts
+test('visual baseline - notes', async ({ page }) => {
+    await page.goto('/s/notes');
+    await expect(page.locator('.sw-topbar')).toBeVisible();
+    await expect(page).toHaveScreenshot('notes.png', {
+        maxDiffPixelRatio: 0.01
+    });
+});
+```
+
+Playwright stores baseline screenshots in `ci/visual-baselines/`.
+`toHaveScreenshot` compares against baseline and produces a diff image
+on mismatch. Foundation for future visual regression gating.
+
+## Sequencing
+
+```
+v0.7.0  Shell Contract        ← prerequisite: surfaces need topbar before
+  │                              runners can assert on it
+  ↓
+v0.7.1  Runner Framework      ← standardize test authoring
+  │
+  ↓
+v0.7.2  Package Runners + CI  ← write the actual tests, wire into CI
+  │
+  ↓
+v0.7.3  Headless E2E          ← automate browser-based runner execution
+```
+
+Each version is independently shippable. v0.7.2's API-based CI gate
+works without v0.7.3's Playwright. v0.7.3 adds coverage for DOM-level
+bugs that API-only runners can't catch.
+
+## Open Questions
+
+1. **Runner package type.** Should `"type": "test-runner"` be a new
+   manifest type, or should runners be `"type": "surface"` with a
+   `"tags": ["test-runner"]` convention? New type is cleaner but requires
+   a `ValidateManifest()` update.
+
+2. **Runner discovery.** The registry surface needs to find all installed
+   runners. Options: (a) scan installed packages for `type: "test-runner"`,
+   (b) runners register themselves via `sw.testing.register()` during SDK
+   boot. Option (a) is declarative and doesn't require runner JS to load
+   before discovery.
+
+3. **Parallel vs sequential.** Should runners execute in parallel?
+   Probably not initially — shared DB state means test isolation is hard.
+   Sequential is safer. Parallel can be a future optimization.
+
+4. **SQLite limitations.** Some runners (cluster, multi-node) are
+   PG-only. The `requires` mechanism should support
+   `"requires_db": "postgres"` for these cases, or runners should
+   self-skip when `sw.config.db_driver === 'sqlite'`.