Feat v0.7.5 e2e ci gate (#59)

ROADMAP.md

@@ -1,14 +1,15 @@
 # Armature — Roadmap
 
-## Current: v0.7.4 — Documentation + Deferred Surface Work
+## Current: v0.7.5 — Headless E2E + CI Gate
 
-Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox,
-storage, realtime, and ops are kernel primitives. Everything else is an extension.
+Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
+sandbox, storage, realtime, and ops are kernel primitives. Everything else
+is an extension.
 
 **Kernel capabilities:** Auth (builtin/mTLS/OIDC) · Users/teams/groups/RBAC ·
 Surfaces/extensions/libraries/workflows · Starlark sandbox (capability-gated) ·
 Object storage (PVC/S3) + ext_data tables · WebSocket hub + realtime pub/sub ·
-Audit log · Notifications · Scheduled tasks
+Audit log · Notifications · Scheduled tasks · Cluster registry + HA
 
 **Completed history:** v0.2.x–v0.5.x fully documented in `CHANGELOG.md`.
 Highlights: RBAC + settings cascade, event bus + triggers, SDK stabilization,
@@ -184,22 +185,282 @@ same pattern as the kernel surface migrations.
 
 | Step | Status | Description |
 |------|--------|-------------|
-| Playwright test harness | | `ci/e2e-surface-test.sh` — docker-compose, chromium, run-all, assert. |
-| Screenshot-on-failure | | Full-page screenshot + console log. CI artifacts. |
-| Navigation smoke test | | Playwright visits every surface. Asserts topbar, no JS errors, home link works. |
-| Visual regression baseline | | Optional screenshot diff. Not a gate — report for review. |
-| CI pipeline integration | | Enable `test-runners` stage, add `e2e-smoke` stage. Failure blocks merge. |
+| CI gating redesign | done | Fix path rules: `VERSION`/`scripts/*` deploy-only, `docs/*` no longer skips deploy, test-runners trigger on any code change. |
+| E2E smoke test harness | done | `ci/e2e-smoke-test.sh` + `ci/e2e-smoke-driver.js` — Playwright visits every surface, asserts topbar, no JS errors, home link. |
+| Screenshot-on-failure | done | Full-page screenshot + console log saved as CI artifacts. |
+| CI pipeline integration | done | Re-enable `test-runners` stage, add `e2e-smoke` stage. Failure blocks merge. |
+| Design docs landed | done | `docs/DESIGN-storage-primitives.md`, `docs/DESIGN-extension-composability.md`. Roadmap expanded through v1.0. |
+
+### v0.7.6 — Code Hygiene + Test Coverage
+
+Codebase analysis (April 2026) found 9 code smells from the chat gut and
+a test coverage gap in the store layer. Fix before v0.8.x kernel expansion.
+
+**Critical Fixes**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| Remove channels from allowedViews | | `db_module.go` — `ext_view_channels` does not exist. `db.view("channels")` crashes. |
+| Fix workflow.html dead routes | | Template calls `/api/v1/channels/{id}/workflow/*` — routes moved to `/api/v1/public/workflows/`. Update template to match. |
+
+**Dead Code Removal**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| Delete SeedTestChannel() | | `database/testhelper.go` — inserts into nonexistent channels table. |
+| Remove RunContext.ChannelID | | `sandbox/runner.go` — vestigial field, unused by any module. |
+| Remove webhook.ChannelID | | `webhook/webhook.go` — vestigial struct field from chat era. |
+| Fix stale comments | | `storage.go` key format, `prometheus.go` route pattern — reference dead `/channels` paths. |
+
+**Migration Hygiene**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| Align SQLite migration numbering | | SQLite 013 = test_runner_type, PG 014 = same. Add placeholder `013_cluster_registry.sql` to SQLite (comment-only, explains PG-only). |
+| Document missing 008 | | Add comment in both 009 files explaining the gap (merged/removed). |
+
+**Test Gap Closure**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| store/ unit tests | | Direct tests for PG + SQLite store implementations. Priority: packages, ext_data, workflows. Target: ≥30% SLOC ratio (from 2%). |
+| workflow/ engine unit tests | | Direct tests for `engine.go`, `routing.go`, `automated.go`. Currently 0% — tested only through handler integration. |
+| middleware/ unit tests | | Auth middleware, rate limit, permissions. Currently 8.9%. |
+| InstallPackage decomposition | | 224 cognitive complexity. Split into: validate → create → DDL → permissions → triggers. Each independently testable. |
+
+### v0.7.7 — API Tokens + Extension Permissions
+
+Personal access tokens (PATs) for programmatic API access, plus
+extension-declared user permissions for backend RBAC enforcement.
+Unblocks: E2E test auth, CI automation, extension permission gating,
+sidecar auth (v0.10.x).
+
+**API Tokens (PATs)**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| `api_tokens` table + migration | | `id`, `user_id`, `name`, `token_hash`, `permissions` (JSON array), `expires_at`, `last_used_at`, `created_at`. PG + SQLite. |
+| Token store interface | | `Create`, `GetByHash`, `ListForUser`, `Revoke`, `CleanExpired`, `UpdateLastUsed`. |
+| Token generation endpoint | | `POST /api/v1/auth/tokens` — name, permissions (subset of user's), expires_at. Returns token once (plain text), stored as SHA-256 hash. |
+| Token management endpoints | | `GET /api/v1/auth/tokens` (list mine), `DELETE /api/v1/auth/tokens/:id` (revoke). |
+| Admin token endpoint | | `POST /api/v1/admin/tokens` — creates token for any user. Explicit admin action, audit logged. |
+| Auth middleware update | | Accept `Bearer arm_pat_...` tokens alongside JWTs. Resolve user + scoped permissions from token. PAT requests skip refresh token flow. |
+| Permission scoping | | Token permissions ⊆ creating user's resolved permissions at creation time. Validated on create. If user loses a permission later, token retains it until revoked (git model). |
+| Settings UI | | Settings → API Tokens tab. Create, list, revoke. Show permissions, expiry, last used. Token value shown once on creation. |
+| Admin UI | | Admin → People → user detail → "Create API Token" action. |
+| E2E test migration | | Replace `ci/e2e-smoke-test.sh` login flow with PAT-based auth. Seed token in docker-compose bootstrap. |
+
+**Extension-Declared User Permissions**
+
+| Step | Status | Description |
+|------|--------|-------------|
+| `user_permissions` manifest field | | Extensions declare permissions users need: `"user_permissions": ["image-gen.use", "image-gen.admin"]`. |
+| Dynamic permission registry | | On install: merge into valid permission set. On uninstall: remove. `AllPermissions` becomes `KernelPermissions + ExtensionPermissions`. |
+| Group UI update | | Admin → Groups → permission checkboxes include extension-registered permissions, grouped by source package. |
+| `permissions` Starlark module | | `permissions.check(user_id, "image-gen.use")` → resolves user's groups, returns bool. New permission: none required (read-only check against kernel data). |
+| `gate_permission` manifest field | | Optional. If set, `ext_api.go` checks this permission before calling `on_request`. Extension doesn't execute for unauthorized users. |
+| `req["permissions"]` in request dict | | `ext_api.go` resolves user permissions and includes them in the request dict. Extensions can check inline without the module. |
 
 ---
 
-## Post-v0.7.x
+## Planned
 
-- **LLM participation** (`llm-bridge` extension)
-- **Rich media extensions:** image generation, code sandbox, STT/TTS
-- **Desktop app** (Tauri or Electron)
-- **Sidecar tier:** container-based extensions
-- **Federation:** cross-instance package sharing
-- **Plugin marketplace** with signing and review
+### v0.8.x — Storage Primitives
+
+Design doc: `docs/DESIGN-storage-primitives.md`
+
+The last major kernel expansion. Completes the primitive set that the
+entire extension ecosystem builds on. After v0.8.x, the kernel is
+feature-complete for 1.0 — all new capabilities are extensions.
+
+**v0.8.0 — `files` Module**
+
+Bridge `ObjectStore` (PVC/S3) into the Starlark sandbox. Extension-scoped
+key namespacing (`ext/{packageID}/`). Permissions: `files.read`,
+`files.write`. Metadata companions stored as sibling objects.
+
+New: `sandbox/files_module.go`. Modified: runner wiring, permission
+constants.
+
+**v0.8.1 — `workspace` Module**
+
+Managed disk directories for tools that need a real filesystem (git,
+compilers, media tools). `workspace.create(name)` → scoped path.
+Permission: `workspace.manage`. Quota enforcement via `WORKSPACE_QUOTA_MB`.
+
+New: `sandbox/workspace_module.go`. Modified: runner wiring, permission
+constants.
+
+**v0.8.2 — Capability Negotiation**
+
+Extensions declare environment requirements in manifest (`capabilities.required`,
+`capabilities.optional`). Kernel validates at install time. Runtime query
+via `settings.has_capability()`. Admin endpoint: `GET /admin/capabilities`.
+
+Detected capabilities: `pgvector`, `workspace`, `object_storage`, `s3`,
+`postgres`.
+
+New: `handlers/capabilities.go`. Modified: install handler, settings module.
+
+**v0.8.3 — Vector Column Type**
+
+`"vector(N)"` in `db_tables` manifest. Maps to native `vector(N)` on
+PG+pgvector, `JSONB` on PG without, `TEXT` on SQLite. New `db.query_similar()`
+with dual-path dispatch (native operator vs Go-side brute force).
+
+Modified: `ext_db_schema.go`, `db_module.go`.
+
+**v0.8.4 — Extension Composability**
+
+Design doc: `docs/DESIGN-extension-composability.md`
+
+Manifest declarations for `slots` (host surfaces declare injection points)
+and `contributes` (extensions declare UI contributions to other packages'
+slots). `lib.require()` relaxed from library-only to any package with
+`exports` — enables full packages (with surfaces, settings, UI) to also
+expose callable backend functions. `sw.slots.renderAll()` SDK helper.
+Admin slot aggregation endpoint.
+
+The composability primitive that enables: LLM tool registration, UI
+contribution (toolbar buttons, image actions), and cross-extension
+function calls. No new tables, migrations, or permissions.
+
+Modified: `sandbox/lib_module.go` (~1 line), `handlers/extensions.go`,
+`src/js/sw/sdk/slots.js`. New: `handlers/admin_slots.go`.
+
+---
+
+### v0.9.x — Reference Extensions
+
+The kernel is complete. This series proves the platform by shipping
+first-party extensions that exercise every primitive. These are
+**extensions, not kernel code** — they ship as `.pkg` files and can be
+uninstalled.
+
+**v0.9.0 — `vector-store` Library**
+
+Library extension: document ingestion, chunk storage, embedding persistence,
+semantic similarity search. Three-tier vector strategy (pgvector → JSONB
+fallback → TEXT fallback). Consumes: `db.write`, `files.read`.
+
+**v0.9.1 — `llm-bridge` Library**
+
+Library extension: model abstraction, tool-use routing, provider BYOK via
+connections. Exposes `complete()`, `embed()`, `classify()`, `register_tool()`
+to consumer extensions. Tool extensions (image-gen, code-exec, web-search)
+register at load time — LLM discovers available tools automatically.
+Consumes: `connections.read`, `api.http`, `db.write`.
+
+**v0.9.2 — `file-share` Extension**
+
+File upload via `api_routes`, storage via `files` module, download links,
+optional team/group ACLs via resource grants. First extension to exercise
+the full `files` primitive end-to-end.
+
+**v0.9.3 — `code-workspace` Extension**
+
+Managed code repositories via `workspace` module. Git clone/pull via
+`api.http` to Gitea/GitHub API. File browser surface. Structural indexing
+(AST-aware) for code navigation. Consumes: `workspace.manage`, `files.write`,
+`api.http`.
+
+**v0.9.4 — `image-gen` + `image-edit` Extensions**
+
+Image generation via external APIs (DALL-E, Stable Diffusion). Registers
+as LLM tool with `llm-bridge`. Contributes regen/edit buttons to
+`chat:image-actions` slot. Image-edit extends with inpaint, outpaint,
+upscale, and style transfer — each contributing action buttons to the
+same slot independently. First full demonstration of the composability
+pattern: three extensions contributing UI to a fourth's surface.
+
+**v0.9.5 — Chat System**
+
+Generic 1-to-N human messaging. `chat-core` library (conversation/message
+CRUD) + `chat` surface. Declares `chat:message-actions`, `chat:image-actions`,
+`chat:composer-tools` slots for extension contribution. LLM participation
+as a follow-on consuming `llm-bridge`. Consumes: `db.write`,
+`realtime.publish`, `files.write` (attachments).
+
+---
+
+### v0.10.x — Sidecar Tier + Polish
+
+**v0.10.0 — Sidecar Tier**
+
+Autonomous out-of-process extensions for workloads that can't run in
+Starlark: ML inference, media transcoding, language servers, local git
+operations. The kernel does NOT manage container lifecycle — sidecars
+are independent processes (Docker, systemd, bare binary) that connect
+inward to the kernel, following the cluster registry pattern:
+
+- Sidecar boots with cluster config (instance URLs, credentials).
+- Connects and self-registers, declaring capabilities ("sklearn",
+  "gpu", "ffmpeg", "sentence-transformers").
+- Goes inert until an admin authorizes (same flow as extension permissions).
+- Kernel dispatches work; sidecar returns results.
+- Heartbeat keeps registration alive; missed heartbeat = eviction.
+
+Communication channel TBD (WebSocket over existing event bus, gRPC, or
+HTTP callback model — design doc required). Authentication via mTLS
+certs or registration tokens.
+
+This model eliminates k8s RBAC dependency for local deployments. The
+kernel stays thin — it accepts registrations and dispatches work. The
+sidecar brings its own compute and manages its own lifecycle.
+
+**v0.10.1 — Native Dialog Audit**
+
+Replace `prompt()`/`confirm()`/`alert()` with `sw.dialog` SDK primitives.
+Accessible, themed, promise-based. Small kernel SDK change that improves
+every surface.
+
+**v0.10.2 — Stability + Migration Tooling**
+
+Proper versioned migrations (post-MVP discipline). `armature migrate`
+CLI command. Backup/restore validation against reference dataset.
+Pre-1.0 schema freeze.
+
+---
+
+### v1.0.0 — Stable Release
+
+The kernel API surface is frozen. Extensions are the product.
+
+Gate criteria:
+
+- All kernel Starlark modules documented with examples
+- All `api_routes` covered by OpenAPI spec
+- Upgrade path tested from v0.8.0 → v1.0.0
+- At least 3 reference extensions shipped and tested (vector-store, llm-bridge, chat)
+- At least 1 cross-extension composability demo (image-gen → chat:image-actions)
+- Headless E2E green on PG + SQLite
+- `armature-ca.sh` + mTLS deployment guide
+- Single-binary + Docker + K8s deployment paths documented
+
+---
+
+## Post-1.0 Horizon
+
+These are candidates, not commitments. Each requires a design doc.
+
+- **Federation** — cross-instance package sharing, identity federation
+- **Package marketplace** — signing, review, discovery registry
+- **Desktop app** — Tauri wrapper for local-first deployment
+- **Offline/sync** — SQLite-first with PG sync for field deployments
+- **Multi-tenant SaaS mode** — tenant isolation at the team boundary
+
+---
+
+## Design Principles
+
+| Principle | Implication |
+|-----------|-------------|
+| Extensions are the product | Chat, tasks, LLM, vector search, file sharing — all extensions. Zero kernel awareness of domain logic. |
+| Kernel stays thin | New kernel primitives require justification. If it can be an extension, it must be. |
+| Progressive enhancement | Every feature works on SQLite. PG adds performance. pgvector adds native vectors. S3 adds scalable storage. |
+| KISS-first | No unnecessary dependencies. Preact+htm (3KB), single binary, dual-DB from one codebase. |
+| Changeset discipline | Each CS independently CI-green. Design docs as implementation contracts. |
+| Pre-1.0 migration freedom | Schema changes fold into existing migrations. Post-1.0: proper versioned migrations. |
 
 ---
 
@@ -228,3 +489,16 @@ same pattern as the kernel surface migrations.
 | Docs is the reference surface | Only surface with shell Topbar, bell, user menu, inline errors. Others converge. |
 | Surface runners over expanding ICD | Different test tier, different failure class. |
 | Headless E2E via Playwright | Runners produce structured JSON; Playwright navigates and reads output. |
+| `db` module is the structured store | No separate KV primitive. Extensions declare tables — infrastructure exists. |
+| `files` module rides ObjectStore | No new kernel tables. Metadata as companion objects. Works on PVC and S3 identically. |
+| `workspace` for real filesystem | Flat blob store can't serve git/compilers/ffmpeg. Managed disk paths are the explicit escape hatch. |
+| Capability negotiation at install | Fail loud with actionable message, not silently at runtime. |
+| Vector column with three-tier fallback | Works everywhere, works fast with pgvector. Documentation > hard gates. |
+| Sidecar deferred to v0.10.x | HTTP module covers external APIs. Local compute is a v0.10.x concern. Sidecars connect inward (like cluster nodes), not spawned outward (no k8s RBAC needed). |
+| No second scripting runtime | Starlark for glue, HTTP for external compute, sidecars for local compute. Three tiers with clear boundaries. Lua/WASM middle tier rejected — Turing-complete enough to be dangerous, not isolated enough to be trustworthy. |
+| Reference extensions prove the platform | First-party `.pkg` files, not kernel code. Uninstallable. Dogfooding. |
+| Composability via slots + exports, not kernel | `sw.slots` + `sw.actions` + `lib.require()` are the composition primitives. Manifest declarations add admin visibility. No inter-extension message bus — too complex, too fragile. |
+| Gut cleanup before kernel expansion | Codebase analysis found 9 smells from the chat gut. Fix before adding new primitives — dead views and broken templates compound when new code builds on them. |
+| PATs over OAuth client credentials | PATs are simpler (one table, one middleware check), cover all use cases (E2E tests, CI, sidecars, user automation), and follow the git hosting convention users already understand. OAuth adds client registration, token exchange, scopes — complexity without benefit for a self-hosted platform. |
+| Extension-declared user permissions | Extensions register custom permissions in the kernel's group system. Admin assigns them to groups. Backend enforces via `permissions.check()` or `gate_permission`. Frontend hides via `sw.can()`. The kernel owns the permission registry; extensions contribute to it. |
+| Admin token as explicit action | Following Venice.ai pattern — admin token creation is a separate endpoint, audit logged, requires `surface.admin.access`. No ambiguity about the privilege level. |