Changeset 0.28.0.1 (#173)
This commit is contained in:
302
docs/archive/AUDIT-0.22.0-PREFLIGHT.md
Normal file
302
docs/archive/AUDIT-0.22.0-PREFLIGHT.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# Pre-Flight Audit — v0.22.0 Kickoff
|
||||
|
||||
**Date:** 2026-03-01
|
||||
**Current version:** 0.21.6
|
||||
**Test instance:** `switchboard-be-test` in `gobha-ai-chat` namespace
|
||||
**Schema:** `012_v0214_git.sql` — up to date
|
||||
|
||||
---
|
||||
|
||||
## 1. Bug: Channel List Corruption (SafeJSON 500)
|
||||
|
||||
### Symptoms
|
||||
|
||||
```
|
||||
23:24:44 ⚠ SafeJSON: marshal failed: json: error calling MarshalJSON for type
|
||||
json.RawMessage: invalid character 'i' looking for beginning of value
|
||||
23:24:44 | 500 | GET "/test/api/v1/channels?page=1&per_page=100&type=direct"
|
||||
```
|
||||
|
||||
One channel in the test database has invalid JSON in its `settings` column. The
|
||||
content starts with the character `i` (likely a bare word like `invalid`,
|
||||
`inactive`, or `in_progress` — not valid JSON). This kills the entire paginated
|
||||
channel list with a 500.
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
**Defense layer 1 — `scanJSON`**: The `ListChannels` scan path at
|
||||
`channels.go:236` correctly uses `scanJSON(&ch.Settings)`, which calls
|
||||
`json.Valid()` and defaults to `{}` on failure. This should catch corrupt data.
|
||||
|
||||
**Defense layer 2 — `SafeJSON`**: Pre-marshals the entire response and returns
|
||||
a clean 500 instead of truncated output. This layer IS firing, meaning layer 1
|
||||
did NOT catch the problem.
|
||||
|
||||
**Inconsistent scan paths**: The Postgres `CreateChannel` RETURNING path at
|
||||
`channels.go:323` scans `&ch.Settings` directly WITHOUT `scanJSON`:
|
||||
|
||||
```go
|
||||
// line 323 — Postgres CreateChannel
|
||||
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,
|
||||
```
|
||||
|
||||
Compare to the SQLite path at line 306 which correctly uses `scanJSON`.
|
||||
|
||||
**How the data got corrupted**: Unknown without DB access. Postgres JSONB
|
||||
columns reject invalid JSON at write time (`::jsonb` cast), so the corruption
|
||||
likely predates the current schema — possibly from a migration, manual DB edit,
|
||||
or an older code version that used TEXT instead of JSONB. Run this to find it:
|
||||
|
||||
```sql
|
||||
SELECT id, title, settings
|
||||
FROM channels
|
||||
WHERE user_id = '664c71e8-cd8b-47fb-b421-9c48abb815ba'
|
||||
AND settings::text NOT LIKE '{%'
|
||||
AND settings::text NOT LIKE '[%'
|
||||
AND settings::text NOT LIKE '"null"';
|
||||
```
|
||||
|
||||
### Fixes Required
|
||||
|
||||
**P0 — Fix the data** (immediate):
|
||||
```sql
|
||||
UPDATE channels
|
||||
SET settings = '{}'::jsonb
|
||||
WHERE settings IS NULL
|
||||
OR NOT (settings::text ~ '^[\[{"]');
|
||||
```
|
||||
|
||||
**P0 — Consistent scanJSON usage** (`channels.go:323`):
|
||||
```go
|
||||
// Before (Postgres CreateChannel):
|
||||
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,
|
||||
|
||||
// After:
|
||||
pq.Array(&tags), scanJSON(&ch.Settings), &ch.CreatedAt, &ch.UpdatedAt,
|
||||
```
|
||||
|
||||
**P1 — Graceful list degradation**: When `SafeJSON` encounters a corrupt item
|
||||
in a paginated list, the current behavior drops the entire response. Instead,
|
||||
add a `SafeMarshalList` helper that:
|
||||
1. Marshals items individually
|
||||
2. Skips corrupt items with a warning log (include the row ID)
|
||||
3. Returns the healthy items with a `warnings` field in the response
|
||||
|
||||
**P1 — Input validation on settings write** (`channels.go:487`):
|
||||
```go
|
||||
if req.Settings != nil {
|
||||
if !json.Valid([]byte(*req.Settings)) {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "settings must be valid JSON"})
|
||||
return
|
||||
}
|
||||
// ... existing merge logic
|
||||
}
|
||||
```
|
||||
|
||||
**P2 — Audit all RawMessage scan sites**: Search for `&ch.Settings`,
|
||||
`&*.Settings`, or any `json.RawMessage` scan that doesn't go through
|
||||
`scanJSON`. Every JSONB column scanned into `json.RawMessage` must use the
|
||||
wrapper.
|
||||
|
||||
### Additional Log Noise
|
||||
|
||||
The git endpoints are returning 400s for workspaces without git repos:
|
||||
|
||||
```
|
||||
23:23:40 | 400 | GET ".../git/status"
|
||||
23:23:56 | 400 | GET ".../git/branches"
|
||||
```
|
||||
|
||||
These are expected (workspace has no `.git` directory), but the frontend should
|
||||
suppress these calls when the workspace has no `git_remote_url` configured.
|
||||
Frontend fix: check `workspace.git_remote_url` before calling git endpoints.
|
||||
|
||||
---
|
||||
|
||||
## 2. CI Variables → Deployment Audit
|
||||
|
||||
Cross-referencing CI secrets/variables (from Gitea) against `backend.yaml`,
|
||||
`config.go`, and runtime logs.
|
||||
|
||||
### Secrets (Gitea Secrets → K8s Secrets)
|
||||
|
||||
| CI Secret | K8s Secret | Key | Status |
|
||||
|-----------|------------|-----|--------|
|
||||
| ENCRYPTION_KEY | switchboard-encryption | ENCRYPTION_KEY | ✅ Active |
|
||||
| POSTGRES_USER | switchboard-db-credentials | POSTGRES_USER | ✅ Active |
|
||||
| POSTGRES_PASSWORD | switchboard-db-credentials | POSTGRES_PASSWORD | ✅ Active |
|
||||
| POSTGRES_ADMIN_USER | — | — | ⚠ Unused by app — likely for CI DB bootstrap |
|
||||
| POSTGRES_ADMIN_PASSWORD | — | — | ⚠ Unused by app — likely for CI DB bootstrap |
|
||||
| S3_ACCESS_KEY | switchboard-s3 | S3_ACCESS_KEY | ✅ Active |
|
||||
| S3_SECRET_KEY | switchboard-s3 | S3_SECRET_KEY | ✅ Active |
|
||||
| SWITCHBOARD_ADMIN_EMAIL | switchboard-admin | email | ✅ Active |
|
||||
| SWITCHBOARD_ADMIN_PASSWORD | switchboard-admin | password | ✅ Active |
|
||||
| SWITCHBOARD_ADMIN_USERNAME | switchboard-admin | username | ✅ Active |
|
||||
|
||||
### Variables (Gitea Variables → envsubst / K8s env)
|
||||
|
||||
| CI Variable | Used In | Value | Status |
|
||||
|-------------|---------|-------|--------|
|
||||
| DOMAIN | ingress.yaml | gobha.ai | ✅ |
|
||||
| NAMESPACE | all manifests | gobha-ai-chat | ✅ |
|
||||
| POSTGRES_HOST | backend.yaml env | postgres-primary.postgres.svc.cluster.local | ✅ |
|
||||
| PROVIDER | ? | venice | ⚠ **Orphan** — not referenced in any manifest or config.go |
|
||||
| S3_BUCKET | switchboard-s3 secret | integration-test | ✅ |
|
||||
| S3_ENDPOINT | switchboard-s3 secret | http://192.168.10.29:7480 | ✅ |
|
||||
| SEED_PROVIDERS | backend.yaml env (direct) | venice:...:micro | ✅ New (Mar 1) |
|
||||
| SEED_USERS | switchboard-seed-users secret | darthvader:...,lukeskywalker:... | ✅ |
|
||||
| STORAGE_BACKEND | backend.yaml env | s3 | ✅ |
|
||||
| STORAGE_CLASS | storage-pvc.yaml | cephfs | ✅ |
|
||||
| STORAGE_SIZE | storage-pvc.yaml | (visible, value cut off) | ✅ |
|
||||
|
||||
### Variables Expected by backend.yaml but NOT in CI Screenshots
|
||||
|
||||
These are likely set in the CI pipeline YAML directly (not as repo variables):
|
||||
|
||||
| Variable | Used In | Expected Source |
|
||||
|----------|---------|-----------------|
|
||||
| ENVIRONMENT | backend.yaml env | Pipeline per-branch |
|
||||
| GIN_MODE | backend.yaml env | Pipeline (release/debug) |
|
||||
| BASE_PATH | backend.yaml env | Pipeline per-deploy (/test, /prod, etc.) |
|
||||
| POSTGRES_PORT | backend.yaml env | Default 5432 (config.go fallback) |
|
||||
| DB_NAME | backend.yaml env | Pipeline per-deploy |
|
||||
| DEPLOY_SUFFIX | all manifests | Pipeline (-test, -prod, etc.) |
|
||||
| BE_IMAGE | backend.yaml image | Pipeline / registry |
|
||||
| IMAGE_TAG | backend.yaml image | Pipeline / git SHA |
|
||||
| BE_REPLICAS | backend.yaml replicas | Pipeline |
|
||||
| BE_MEMORY_REQUEST | backend.yaml resources | Pipeline |
|
||||
| BE_MEMORY_LIMIT | backend.yaml resources | Pipeline |
|
||||
| BE_CPU_REQUEST | backend.yaml resources | Pipeline |
|
||||
| BE_CPU_LIMIT | backend.yaml resources | Pipeline |
|
||||
| EXTRACTION_CONCURRENCY | backend.yaml env | Pipeline (default 3) |
|
||||
|
||||
### Issues Found
|
||||
|
||||
1. **`PROVIDER` variable is orphaned.** Value is "venice" but nothing in
|
||||
backend.yaml or config.go references it. Likely a legacy variable from
|
||||
before `SEED_PROVIDERS` was added. Clean up or document.
|
||||
|
||||
2. **`SEED_PROVIDERS` contains an API key in a CI Variable (not Secret).**
|
||||
The screenshot shows the Venice API key visible in plaintext in the Variables
|
||||
section. This should be moved to Gitea Secrets for proper masking. The
|
||||
backend.yaml needs a corresponding mapping (currently SEED_PROVIDERS may be
|
||||
injected via envsubst or direct env, verify the CI pipeline).
|
||||
|
||||
3. **`SEED_USERS` passwords are plaintext in CI.** `password1234` for both
|
||||
test users. Acceptable for test environment but should be documented as
|
||||
test-only. Production pipeline should NOT have SEED_USERS set at all (and
|
||||
`SeedProviders` already blocks production).
|
||||
|
||||
4. **Missing `SEED_PROVIDERS` in backend.yaml env block.** The K8s manifest
|
||||
doesn't have a `SEED_PROVIDERS` env var entry. It works because the CI
|
||||
pipeline likely injects it via envsubst into a direct env entry, but this
|
||||
should be made explicit in backend.yaml for documentation parity.
|
||||
|
||||
5. **No `JWT_SECRET` in CI secrets.** Config.go defaults to
|
||||
`"dev-secret-change-me"`. The test instance may be running with this
|
||||
default. Add a proper secret for test environments.
|
||||
|
||||
6. **No `S3_REGION` or `S3_PREFIX` in CI.** Config.go defaults to
|
||||
`us-east-1` and `""` respectively. Fine for Ceph RGW but should be
|
||||
documented.
|
||||
|
||||
### Recommendations
|
||||
|
||||
- Move `SEED_PROVIDERS` from CI Variable to CI Secret
|
||||
- Remove or document the orphaned `PROVIDER` variable
|
||||
- Add `JWT_SECRET` to CI Secrets and wire through backend.yaml
|
||||
- Add `SEED_PROVIDERS` env entry to backend.yaml (with secretKeyRef)
|
||||
- Add a CI pipeline comment documenting which variables are set inline
|
||||
vs. in repo settings
|
||||
|
||||
---
|
||||
|
||||
## 3. Test Hardening — Unhappy Paths
|
||||
|
||||
Current test coverage is strong on the happy path (164+ integration tests).
|
||||
The corruption bug exposes gaps in unhappy-path coverage.
|
||||
|
||||
### Missing Test Categories
|
||||
|
||||
**A. Data corruption resilience** (integration_test.go):
|
||||
|
||||
- [ ] **Corrupt JSONB in channels**: INSERT a channel with valid settings, then
|
||||
UPDATE settings directly via SQL with invalid text. Verify GET /channels
|
||||
still returns 200 with the corrupt channel included (settings defaulted to
|
||||
`{}`).
|
||||
- [ ] **Corrupt JSONB in projects**: Same pattern for project settings.
|
||||
- [ ] **Corrupt JSONB in teams**: Same for team settings/policies.
|
||||
- [ ] **NULL settings column**: Verify channels with NULL settings serialize
|
||||
correctly (should be `{}` or `null`, not crash).
|
||||
- [ ] **Corrupt tags column**: Insert channel with malformed tags array.
|
||||
Verify list doesn't 500.
|
||||
|
||||
**B. Settings write validation**:
|
||||
|
||||
- [ ] **Invalid JSON in channel settings update**: PUT /channels/:id with
|
||||
`{"settings": "not-json"}` — should 400, not write to DB.
|
||||
- [ ] **Oversized settings**: PUT with a 10MB settings blob — should be
|
||||
rejected.
|
||||
- [ ] **SQL injection via settings merge**: PUT with settings containing SQL
|
||||
metacharacters — verify parameterized query prevents injection.
|
||||
|
||||
**C. Scan alignment tests** (safe_json_test.go):
|
||||
|
||||
- [ ] **Column count mismatch**: Mock a query result with wrong column count,
|
||||
verify clean error instead of panic.
|
||||
- [ ] **Type mismatch**: Scan a TEXT column into json.RawMessage without
|
||||
scanJSON — verify SafeJSON catches it.
|
||||
- [ ] **Driver buffer reuse**: Scan multiple rows with RawMessage, verify each
|
||||
row's data is independent (copy, not reference).
|
||||
|
||||
**D. Startup resilience** (config/seed tests):
|
||||
|
||||
- [ ] **SEED_PROVIDERS with invalid format**: `"openai"` (no key), `":"`,
|
||||
`":::"`, empty string — verify graceful skip.
|
||||
- [ ] **SEED_PROVIDERS with unknown provider**: `"unknown:key123"` — verify
|
||||
warning log, not crash.
|
||||
- [ ] **SEED_USERS with invalid format**: Malformed CSV — verify skip.
|
||||
- [ ] **DATABASE_URL with unreachable host**: Verify startup logs error and
|
||||
serves 503 on API endpoints.
|
||||
|
||||
**E. WebSocket reconnection**:
|
||||
|
||||
- [ ] **Rapid disconnect/reconnect**: The logs show multiple ws connect/
|
||||
disconnect cycles in quick succession. Verify no goroutine leaks,
|
||||
no duplicate event delivery.
|
||||
- [ ] **Auth token expiry during active connection**: Verify clean disconnect
|
||||
with appropriate error, not silent data loss.
|
||||
|
||||
**F. Frontend error handling** (manual or e2e):
|
||||
|
||||
- [ ] **500 on channel list**: Frontend should show error state, not blank
|
||||
screen. Currently unknown behavior.
|
||||
- [ ] **400 on git endpoints for non-git workspace**: Frontend should not
|
||||
call git endpoints when workspace has no git_remote_url.
|
||||
- [ ] **Network timeout during streaming completion**: Verify partial
|
||||
response is preserved, user can retry.
|
||||
|
||||
### Test Infrastructure Needed
|
||||
|
||||
1. **SQL fixture injection**: Helper function to INSERT corrupt data directly
|
||||
via SQL, bypassing store layer validation. For corruption resilience tests.
|
||||
|
||||
2. **SafeJSON audit script**: `grep -rn '&.*\.Settings\|json.RawMessage'` to
|
||||
find all RawMessage scan sites. Run as CI lint check to catch regressions.
|
||||
|
||||
3. **Integration test for each migration**: Both fresh-install path AND
|
||||
upgrade path from previous version. Currently tested manually — automate.
|
||||
|
||||
---
|
||||
|
||||
## 4. Quick Wins Before v0.22.0 Work Starts
|
||||
|
||||
Priority-ordered items that should land as v0.21.7 patches:
|
||||
|
||||
1. **Fix corrupt data in test DB** — SQL UPDATE (5 min)
|
||||
2. **Add scanJSON to CreateChannel Postgres path** — one-line fix (5 min)
|
||||
3. **Add json.Valid check on settings write** — channels.go UpdateChannel (10 min)
|
||||
4. **Move SEED_PROVIDERS to CI Secret** — CI UI change (5 min)
|
||||
5. **Suppress git endpoint calls for non-git workspaces** — frontend (15 min)
|
||||
6. **Add corruption resilience integration tests** — 2-3 tests (1 hr)
|
||||
7. **Add JWT_SECRET to CI Secrets** — CI + backend.yaml (15 min)
|
||||
1153
docs/archive/DESIGN-0.24.0.md
Normal file
1153
docs/archive/DESIGN-0.24.0.md
Normal file
File diff suppressed because it is too large
Load Diff
394
docs/archive/DESIGN-0.26.0.md
Normal file
394
docs/archive/DESIGN-0.26.0.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# DESIGN — v0.26.0: Workflow Engine
|
||||
|
||||
**Status:** Shipped (v0.26.0–v0.26.5, 15 changesets)
|
||||
**Branch:** `0.26.0`
|
||||
**Scope:** Team-owned staged processes, AI intake, human assignment, visitor experience
|
||||
**Depends on:** Dynamic surfaces (v0.25.0), multi-participant channels (v0.23.x),
|
||||
anonymous identity (v0.24.3), permissions (v0.24.2)
|
||||
|
||||
---
|
||||
|
||||
## Phasing
|
||||
|
||||
Six sub-releases. Each is shippable and testable independently.
|
||||
Deferred tech debt from v0.21–v0.25 absorbed into v0.26.0 (phase 0).
|
||||
|
||||
```
|
||||
v0.26.0 Foundation: cleanup + context-aware tool system
|
||||
│
|
||||
v0.26.1 Workflow definitions + versioning (schema + CRUD)
|
||||
│
|
||||
v0.26.2 Workflow instances + stage transitions (runtime)
|
||||
│
|
||||
v0.26.3 Visitor experience (surfaces + branded chat)
|
||||
│
|
||||
v0.26.4 AI intake + assignment queue
|
||||
│
|
||||
v0.26.5 Team collaboration + workflow builder UI
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## v0.26.0 — Foundation
|
||||
|
||||
Cleanup debt, then build the tool infrastructure that workflows need.
|
||||
|
||||
### Deferred Cleanup (absorb from v0.21–v0.25)
|
||||
|
||||
**Session cleanup job**
|
||||
- Background goroutine: delete `session_participants` older than N days
|
||||
with no associated messages
|
||||
- Configurable via `SESSION_EXPIRY_DAYS` env var (default: 30)
|
||||
- Runs on startup + every 6h
|
||||
- Closes v0.24.3 deferred item
|
||||
|
||||
**Stale code TODOs**
|
||||
- `surfaces.go:108-109` — surface delete: clean up static assets + templates
|
||||
from `SURFACE_ASSET_DIR/{id}/` and `SURFACE_TEMPLATE_DIR/{id}/`
|
||||
- `completion.go:164` — remove misleading TODO comment (pricing IS populated
|
||||
in `recordUsage()` at line ~1996)
|
||||
- `intrinsic.go:19` — remove stale "TODO: 0.9.2" comment (shipped in v0.22.0)
|
||||
|
||||
**Roadmap housekeeping**
|
||||
- Mark shipped items in "Technical Debt + Deferred Items" section:
|
||||
- Rate limit tracking (v0.22.4)
|
||||
- Auto-disable policy (v0.22.4)
|
||||
- Tool health recording (v0.22.4)
|
||||
- PDF export via pandoc (v0.22.4)
|
||||
- Persona tool grant enforcement (v0.25.0)
|
||||
- Clean up stale deferred-into version markers
|
||||
|
||||
### Context-Aware Tool System
|
||||
|
||||
_Absorbed from v0.25.0 roadmap. Prerequisite for workflow tool scoping._
|
||||
|
||||
**`ToolContext` struct** (`tools/types.go`)
|
||||
- Fields: `ChannelType`, `WorkspaceID`, `WorkflowID`, `TeamID`,
|
||||
`PersonaID`, `IsVisitor`
|
||||
- Populated from channel record in completion handler before tool execution
|
||||
|
||||
**`Require` predicates**
|
||||
- `Availability() Require` method on `Tool` interface
|
||||
- `BaseTool` embed: defaults to `AlwaysAvailable` (backward compat)
|
||||
- Built-in predicates: `RequireWorkspace`, `RequireWorkflow`,
|
||||
`RequireTeam`, `DenyVisitor`, `All()`
|
||||
- `tools.AvailableFor(tctx, disabled)` replaces `AllDefinitionsFiltered()`
|
||||
|
||||
**Self-declaring tools**
|
||||
- Workspace tools: `RequireWorkspace + DenyVisitor`
|
||||
- Git tools: `RequireWorkspace + DenyVisitor`
|
||||
- `workspace_create`: available only when no workspace bound
|
||||
- Memory/notes tools: `DenyVisitor`
|
||||
- Eliminates `WorkspaceToolNames()` / `GitToolNames()` manual suppression
|
||||
|
||||
**`ExecutionContext` extension**
|
||||
- Add `WorkflowID`, `TeamID` to existing `ExecutionContext`
|
||||
- Populated from channel record in completion handler
|
||||
|
||||
**Persona tool grant enforcement hardening**
|
||||
- v0.25.0 shipped the second-pass allowlist in completion.go:917
|
||||
- For workflows: version snapshot must include persona tool grants
|
||||
at snapshot time (frozen for running instances)
|
||||
|
||||
### Route Namespaces
|
||||
|
||||
Two distinct namespaces. No collision.
|
||||
|
||||
**`/s/:slug` — Extension Surfaces (application extensions)**
|
||||
- Single-page, self-contained sub-applications
|
||||
- Examples: custom dashboard, kanban board, form builder
|
||||
- One template, one JS entry point, scoped CSS
|
||||
- Registered via surface manifest (v0.25.0 `surface_registry`)
|
||||
|
||||
**`/w/:scope/:slug` — Workflows (business logic automation)**
|
||||
- `:scope` is `team-slug` or `global` — workflows belong to a team or
|
||||
are org-wide. Prevents slug collisions across teams.
|
||||
- Multi-page: landing, visitor chat, team review, admin tracking —
|
||||
different views depending on who is accessing (visitor vs. team
|
||||
member vs. admin)
|
||||
- Involves customers and team members, possibly spanning multiple teams
|
||||
- Purpose-built Go templates per view, not a single-page pattern
|
||||
|
||||
Both use Go template rendering. They share `base.html` but have
|
||||
independent template trees and JS entry points.
|
||||
|
||||
### Migration
|
||||
|
||||
- 023_v0260_foundation.sql: session cleanup additions (if schema changes
|
||||
needed), no new tables in this phase
|
||||
|
||||
---
|
||||
|
||||
## v0.26.1 — Workflow Definitions + Versioning
|
||||
|
||||
Schema and CRUD for defining workflows. No runtime yet.
|
||||
|
||||
### Tables
|
||||
|
||||
**`workflows`**
|
||||
- `id`, `team_id` (nullable — NULL = global), `name`,
|
||||
`slug` (unique within scope: `UNIQUE(team_id, slug)` with
|
||||
partial index for global), `description`
|
||||
- `branding` (JSONB: accent color, logo URL, tagline)
|
||||
- `entry_mode` (enum: `public_link`, `team_only`)
|
||||
- `is_active`, `version` (auto-increment on edit)
|
||||
- `on_complete` (JSONB, nullable — v0.27.0 chaining hook, NULL for now.
|
||||
Future schema: `{"action": "start_workflow", "target_slug": "...",
|
||||
"data_map": {...}}`. Column exists early so the table doesn't need
|
||||
migration when chaining lands.)
|
||||
- `retention` (JSONB — `{"mode": "archive"|"delete", "delete_after_days": N}`.
|
||||
Default: `{"mode": "archive"}`)
|
||||
- `created_by`, `created_at`, `updated_at`
|
||||
|
||||
**`workflow_stages`**
|
||||
- `id`, `workflow_id`, `ordinal`, `name`
|
||||
- `persona_id` (FK — persona drives this stage)
|
||||
- `assignment_team_id` (FK — which team handles human review)
|
||||
- `form_template` (JSONB — fields the persona should collect)
|
||||
- `history_mode` (enum: `full`, `summary`, `fresh` — default `full`)
|
||||
- `auto_transition` (bool — advance automatically when form complete)
|
||||
- `transition_rules` (JSONB — conditions, round-robin config)
|
||||
|
||||
**`workflow_versions`**
|
||||
- `id`, `workflow_id`, `version_number`
|
||||
- `snapshot` (JSONB — full serialized definition + stages + tool grants)
|
||||
- `created_at`
|
||||
|
||||
### API
|
||||
|
||||
- Team-admin CRUD: create/edit/delete workflows and stages
|
||||
- Reorder stages (PATCH ordinal)
|
||||
- `workflow.create` permission required
|
||||
- Publish action: snapshot current definition → `workflow_versions`
|
||||
- Slug validation: lowercase, alphanumeric + hyphens, unique within scope
|
||||
(same slug can exist under different teams). URL resolves as
|
||||
`/w/team-slug/workflow-slug` or `/w/global/workflow-slug`.
|
||||
|
||||
### Design Decisions (to flesh out)
|
||||
|
||||
- Branding schema: minimal (accent + logo + tagline) or extensible JSONB?
|
||||
- Stage persona: must be team-scoped persona? Or any accessible persona?
|
||||
|
||||
### Migration
|
||||
|
||||
- 023_v0261_workflows.sql: `workflows`, `workflow_stages`, `workflow_versions`
|
||||
|
||||
---
|
||||
|
||||
## v0.26.2 — Workflow Instances + Stage Transitions
|
||||
|
||||
Channels become workflow runtime containers.
|
||||
|
||||
### Channel Extensions
|
||||
|
||||
- `workflow_id`, `workflow_version` columns on `channels`
|
||||
(populated on workflow channel creation)
|
||||
- `current_stage` (ordinal), `stage_data` (JSONB)
|
||||
- `workflow_status` enum: `active`, `completed`, `stale`, `cancelled`
|
||||
- `last_activity_at` tracking
|
||||
|
||||
### Entry Point
|
||||
|
||||
- Public link: `/w/:scope/:slug` → creates workflow channel, adds anonymous
|
||||
session participant (v0.24.3), binds stage 1 persona, auto-creates
|
||||
workspace. `:scope` is team slug or `global`.
|
||||
- Team-only: same flow but requires authenticated user
|
||||
|
||||
### Stage Transitions
|
||||
|
||||
- AI-triggered: `workflow_advance` tool (see v0.26.4)
|
||||
- Human-triggered: button in channel header
|
||||
- Rejection: return to previous stage with reason text
|
||||
- On transition: swap active persona, notify assignment team,
|
||||
update channel metadata, create channel-scoped note with collected data
|
||||
|
||||
**History mode** (per-stage, configurable in `workflow_stages`):
|
||||
- `full` — new persona sees complete conversation history with a system
|
||||
boundary message (same pattern as @mention context boundaries, v0.23.0).
|
||||
Simplest to implement, best for continuity-sensitive workflows.
|
||||
- `summary` — utility-role summarization of prior stages injected as a
|
||||
system message. New persona starts with context but not raw history.
|
||||
- `fresh` — new persona sees nothing from prior stages. Clean slate.
|
||||
Simplest for independent review stages.
|
||||
|
||||
Default: `full`. Stored as `history_mode` enum on `workflow_stages`.
|
||||
|
||||
### Staleness Sweep
|
||||
|
||||
- Background goroutine (like health accumulator pattern)
|
||||
- Marks idle instances as `stale` after configurable threshold
|
||||
- Stale UX: visitor sees "Continue or Start Over"
|
||||
(start over = new instance on latest version)
|
||||
|
||||
### Migration
|
||||
|
||||
- Adds columns to `channels`, or separate `workflow_instances` table
|
||||
(TBD — inline columns simpler, separate table cleaner)
|
||||
|
||||
---
|
||||
|
||||
## v0.26.3 — Visitor Experience
|
||||
|
||||
Purpose-built surfaces for anonymous workflow participants.
|
||||
|
||||
### Landing Page
|
||||
|
||||
- `GET /w/:scope/:slug` — branded page with persona avatar, description, "Start"
|
||||
- Go template: `workflow-landing.html`
|
||||
- Reads `workflow.branding` JSONB for accent color, logo, tagline
|
||||
- System dark/light mode (no theme toggle — visitors don't have prefs)
|
||||
|
||||
### Visitor Chat
|
||||
|
||||
- `GET /w/:scope/:slug/c/:channelId` — bubble chat (NOT full ChatPane)
|
||||
- `workflow-chat.js`: lightweight WebSocket client
|
||||
- Message send/receive
|
||||
- Markdown rendering (marked.js + DOMPurify, already available)
|
||||
- Typing indicator
|
||||
- Stage transition animation (visual feedback on advance)
|
||||
- Scoped: no sidebar, no settings, no navigation
|
||||
- Session auth via `AuthOrSession` middleware (v0.24.3)
|
||||
|
||||
### Tool Scoping
|
||||
|
||||
- `DenyVisitor` predicates prevent visitors from accessing workspace,
|
||||
git, memory, notes tools
|
||||
- Persona tool grants further restrict per-stage
|
||||
|
||||
### Design Decisions (to flesh out)
|
||||
|
||||
- Visitor can upload files? (attachments to workflow channel)
|
||||
- Visitor can see previous stage history? (probably not — each stage
|
||||
is a clean persona conversation)
|
||||
- Mobile-first layout for visitor surfaces
|
||||
|
||||
---
|
||||
|
||||
## v0.26.4 — AI Intake + Assignment Queue
|
||||
|
||||
The AI does the data collection. Humans review and act.
|
||||
|
||||
### `workflow_advance` Tool
|
||||
|
||||
- `RequireWorkflow` availability predicate
|
||||
- Input: collected form data (JSONB matching `form_template`)
|
||||
- Validates all required fields present
|
||||
- Triggers stage transition (same path as human-triggered)
|
||||
- Creates channel-scoped note with structured form response
|
||||
|
||||
### Form Template → System Prompt
|
||||
|
||||
- Stage `form_template` JSONB injected into persona system prompt:
|
||||
"Collect the following information from the visitor: [field list]"
|
||||
- Persona conducts conversational intake
|
||||
- When all fields gathered → calls `workflow_advance` with data
|
||||
- Rejection: persona sees rejection reason in conversation, re-collects
|
||||
|
||||
### Assignment Queue
|
||||
|
||||
**`workflow_assignments` table**
|
||||
- `id`, `channel_id`, `stage` (ordinal), `team_id`
|
||||
- `assigned_to` (nullable user_id), `status` (unassigned/claimed/completed)
|
||||
- `created_at`, `claimed_at`, `completed_at`
|
||||
|
||||
**Claim model**
|
||||
- Unassigned workflow channels visible to all team members
|
||||
- Optimistic lock: `UPDATE ... WHERE status = 'unassigned'`
|
||||
- Round-robin auto-assignment (optional, per-stage `transition_rules`)
|
||||
|
||||
**Notifications**
|
||||
- New workflow assignment → notification to team (existing infra)
|
||||
- WebSocket push to online team members
|
||||
- Claim confirmation notification to claimer
|
||||
|
||||
### Migration
|
||||
|
||||
- `workflow_assignments` table
|
||||
|
||||
---
|
||||
|
||||
## v0.26.5 — Team Collaboration + Workflow Builder UI
|
||||
|
||||
Frontend surfaces for building and managing workflows.
|
||||
|
||||
### Team Member View
|
||||
|
||||
- Assigned member sees full history (AI intake + artifacts + notes)
|
||||
- Persona remains active — assists both visitor and team member
|
||||
- Member can: advance stage, reject (with reason), add notes,
|
||||
invoke tools, reassign, escalate
|
||||
|
||||
### Workflow Builder
|
||||
|
||||
- Team admin panel section
|
||||
- Stage list editor: add/remove/reorder stages
|
||||
- Per-stage config: persona picker, assignment team, form template editor
|
||||
- Branding editor: accent color, logo upload, tagline
|
||||
- Publish button (creates version snapshot)
|
||||
- Read-only tool grants display per stage persona
|
||||
|
||||
### Channel Header (Workflow Mode)
|
||||
|
||||
- Workflow stage indicator (step N of M, stage name)
|
||||
- Advance / Reject buttons (permission-gated)
|
||||
- Assignment info (who claimed, when)
|
||||
- History mode indicator (full/summary/fresh)
|
||||
|
||||
### Queue UI
|
||||
|
||||
Prototype both approaches in v0.26.5, decide based on usage:
|
||||
|
||||
**Option A: Sidebar section** for team members — shows unassigned workflow
|
||||
channels for user's teams, count badge, click opens in main pane.
|
||||
Low friction, always visible.
|
||||
|
||||
**Option B: Dedicated surface** (`/workflows` or team-admin view) for
|
||||
tracking and management — full table with filters, assignment history,
|
||||
metrics. Better for high-volume operations.
|
||||
|
||||
Likely outcome: both. Sidebar for team members (day-to-day claims),
|
||||
dedicated surface for team admins (tracking, reporting, bulk ops).
|
||||
|
||||
---
|
||||
|
||||
## Resolved Decisions
|
||||
|
||||
1. **Workflow channel lifecycle.** Configurable per-workflow retention
|
||||
policy. Default: archive (read-only, `ai_mode='off'`, no new messages).
|
||||
Optional: auto-delete after N days. Stored as `retention` JSONB on
|
||||
`workflows` table.
|
||||
|
||||
2. **Multi-stage persona conversations.** Three modes, configurable
|
||||
per-stage via `history_mode` enum on `workflow_stages`:
|
||||
- `full` — complete history + boundary message (default, simplest)
|
||||
- `summary` — utility-role summarization injected as system message
|
||||
- `fresh` — clean slate, no prior context
|
||||
First and third are simplest to implement; ship those first,
|
||||
`summary` can land as a follow-up.
|
||||
|
||||
3. **Visitor re-entry.** Resume if session cookie matches and instance
|
||||
is `active`. Otherwise start new on latest version.
|
||||
|
||||
4. **Workflow-to-workflow chaining.** Deferred to v0.27.0 (tasks/agents).
|
||||
`on_complete` JSONB column on `workflows` table ships as nullable in
|
||||
v0.26.1 migration so the schema is pre-wired. v0.27.0 populates it
|
||||
and adds the trigger logic. No migration needed when chaining lands.
|
||||
|
||||
5. **Route namespaces.** `/s/:slug` for extension surfaces (single-page
|
||||
sub-apps). `/w/:scope/:slug` for workflows (multi-page, role-dependent
|
||||
views). No collision. See "Route Namespaces" in v0.26.0.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Cutting Concerns
|
||||
|
||||
**Testing strategy:** Integration tests per phase. v0.26.0 tool context
|
||||
tests. v0.26.1 workflow CRUD + versioning tests. v0.26.2 stage transition
|
||||
tests. v0.26.4 assignment claim concurrency test.
|
||||
|
||||
**Migration numbering:** Single migration per phase (023, 024, ...) or
|
||||
consolidated? Leaning toward: one per phase, squash at tag time if needed.
|
||||
|
||||
**Backward compat:** Non-workflow channels unaffected. `ToolContext` with
|
||||
`BaseTool` defaults means existing tools work without changes. Workflow
|
||||
columns on `channels` are nullable.
|
||||
842
docs/archive/DESIGN-0.27.0.md
Normal file
842
docs/archive/DESIGN-0.27.0.md
Normal file
@@ -0,0 +1,842 @@
|
||||
# DESIGN — v0.27.0+: Debt Clearance, Extension Surfaces, Tasks
|
||||
|
||||
**Status:** Draft
|
||||
**Scope:** Deferred debt clearance (v0.27.0), Tasks / Autonomous Agents (v0.27.x), TBD pull-forward (v0.28.0)
|
||||
**Depends on:** Workflow Engine (v0.26.0), Dynamic Surfaces (v0.25.0), Permissions (v0.24.2)
|
||||
**Current version:** 0.26.5
|
||||
|
||||
---
|
||||
|
||||
## Versioning Plan
|
||||
|
||||
```
|
||||
v0.27.0 Debt Clearance: Extension Surface Routes + Workflow Polish
|
||||
│
|
||||
v0.27.1 Tasks Foundation: Service Channels + Scheduler
|
||||
│
|
||||
v0.27.2 Task Execution: Budgets + Admin Controls
|
||||
│
|
||||
v0.27.3 Task Chaining: Webhooks + Workflow-to-Workflow
|
||||
│
|
||||
v0.27.4 Personal Tasks: BYOK Scheduling + User Task UI
|
||||
│
|
||||
v0.28.0 Platform Polish (TBD pull-forward)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Audit — Items Already Shipped
|
||||
|
||||
The ROADMAP lists these as deferred but the code proves they're done:
|
||||
|
||||
| Item | Evidence | Listed as deferred in |
|
||||
|------|----------|----------------------|
|
||||
| Persona tool grant enforcement in completion handler | `completion.go:928–949` — second-pass allowlist via `GetToolGrants()` | v0.25.0, v0.26.0 |
|
||||
| Version snapshot includes persona tool grants | `workflows.go:263–273` — `stageSnapshot.ToolGrants` populated at publish | v0.25.0 |
|
||||
| Session cleanup job | Shipped in v0.26.0 (background goroutine, `SESSION_EXPIRY_DAYS`) | v0.24.3 |
|
||||
|
||||
**Action:** Cross off all three in the ROADMAP's deferred sections and in
|
||||
the v0.25.0 / v0.26.0 milestone checklists.
|
||||
|
||||
---
|
||||
|
||||
## v0.27.0 — Debt Clearance
|
||||
|
||||
All deferred items from v0.25.0 and v0.26.0 plus outstanding tech debt
|
||||
that blocks future work. No new features — just finishing what's owed.
|
||||
|
||||
### Phase 1: Extension Surface Routes (`/s/:slug`)
|
||||
|
||||
The longest-deferred item (originally v0.21.3). Infrastructure exists:
|
||||
`surface_registry` table, `InstallSurface` handler (zip upload + asset
|
||||
extraction), `ListEnabledSurfaces` API. What's missing is the runtime
|
||||
route mounting, template rendering, and frontend nav integration.
|
||||
|
||||
**Problem: Hardcoded surface conditionals in `base.html`**
|
||||
|
||||
Lines 61–66 and 118–122 of `base.html` use `{{if eq .Surface "chat"}}`
|
||||
chains. Extension surfaces fall through to "Unknown surface". Same
|
||||
pattern for CSS includes (line 25–26) and script includes (118–122).
|
||||
|
||||
**Solution: Generic extension surface template + dynamic blocks**
|
||||
|
||||
```
|
||||
base.html changes:
|
||||
{{if eq .Surface "chat"}}...
|
||||
{{else if eq .Surface "admin"}}...
|
||||
...
|
||||
{{else if .Manifest}}{{template "surface-extension" .}}
|
||||
{{else}}<div>Unknown surface: {{.Surface}}</div>
|
||||
{{end}}
|
||||
```
|
||||
|
||||
New template `templates/surfaces/extension.html`:
|
||||
```html
|
||||
{{define "surface-extension"}}
|
||||
<div id="extension-surface" class="extension-surface"
|
||||
data-surface-id="{{.Surface}}"
|
||||
data-manifest='{{.Manifest | toJSON}}'>
|
||||
{{/* Extension JS mounts into this container */}}
|
||||
<div id="extension-mount"></div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{define "css-extension"}}
|
||||
{{/* Extension CSS loaded dynamically from surfacesDir */}}
|
||||
{{if .Manifest}}
|
||||
<link rel="stylesheet" href="{{.BasePath}}/surfaces/{{.Surface}}/css/main.css?v={{.Version}}">
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{define "scripts-extension"}}
|
||||
{{if .Manifest}}
|
||||
<script nonce="{{.CSPNonce}}" src="{{.BasePath}}/surfaces/{{.Surface}}/js/main.js?v={{.Version}}"></script>
|
||||
{{end}}
|
||||
{{end}}
|
||||
```
|
||||
|
||||
**Dynamic route registration at startup:**
|
||||
|
||||
```go
|
||||
// pages.go — New() after registerCoreSurfaces():
|
||||
func (e *Engine) loadExtensionSurfaces() {
|
||||
if e.stores.Surfaces == nil {
|
||||
return
|
||||
}
|
||||
ctx := context.Background()
|
||||
surfaces, err := e.stores.Surfaces.List(ctx)
|
||||
if err != nil {
|
||||
log.Printf("[pages] Failed to load extension surfaces: %v", err)
|
||||
return
|
||||
}
|
||||
for _, sr := range surfaces {
|
||||
if sr.Source == "core" {
|
||||
continue // Core surfaces already registered
|
||||
}
|
||||
route, _ := sr.Manifest["route"].(string)
|
||||
if route == "" {
|
||||
route = "/s/" + sr.ID // Default to /s/:id
|
||||
}
|
||||
manifest := SurfaceManifest{
|
||||
ID: sr.ID,
|
||||
Route: route,
|
||||
Title: sr.Title,
|
||||
Template: "surface-extension",
|
||||
Auth: authFromManifest(sr.Manifest), // default "authenticated"
|
||||
Layout: layoutFromManifest(sr.Manifest),
|
||||
Source: "extension",
|
||||
}
|
||||
e.surfaces = append(e.surfaces, manifest)
|
||||
}
|
||||
log.Printf("[pages] Loaded %d extension surfaces from registry",
|
||||
len(surfaces) - countCore(surfaces))
|
||||
}
|
||||
```
|
||||
|
||||
**nginx config — serve extension static assets:**
|
||||
|
||||
```nginx
|
||||
# Extension surface assets
|
||||
location /surfaces/ {
|
||||
alias /data/surfaces/;
|
||||
expires 1h;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
```
|
||||
|
||||
**Frontend nav integration:**
|
||||
|
||||
`ListEnabledSurfaces` API already returns route + title for all enabled
|
||||
surfaces. The sidebar/nav needs to render extension surfaces as
|
||||
additional items. Extension surfaces appear in a "Surfaces" section
|
||||
of the sidebar, below the core nav items.
|
||||
|
||||
```js
|
||||
// app.js or pages.js — on init:
|
||||
const surfaces = await App.api.get('/api/v1/surfaces');
|
||||
surfaces.filter(s => !['chat','admin','settings','editor','notes'].includes(s.id))
|
||||
.forEach(s => renderExtensionNavItem(s));
|
||||
```
|
||||
|
||||
**Extension surface manifest contract (`.surface` archive):**
|
||||
|
||||
```
|
||||
my-dashboard.surface (zip)
|
||||
├── manifest.json
|
||||
├── js/
|
||||
│ └── main.js ← entry point, mounts into #extension-mount
|
||||
├── css/
|
||||
│ └── main.css ← scoped styles
|
||||
└── assets/ ← images, fonts, etc.
|
||||
```
|
||||
|
||||
`manifest.json`:
|
||||
```json
|
||||
{
|
||||
"id": "my-dashboard",
|
||||
"title": "Dashboard",
|
||||
"route": "/s/dashboard",
|
||||
"auth": "authenticated",
|
||||
"layout": "single",
|
||||
"components": ["chat-pane"],
|
||||
"hooks": ["surface"]
|
||||
}
|
||||
```
|
||||
|
||||
The extension JS receives the standard `ctx` object (same extension API
|
||||
from v0.11.0) plus access to platform components:
|
||||
|
||||
```js
|
||||
// main.js — extension surface entry point
|
||||
(function() {
|
||||
const mount = document.getElementById('extension-mount');
|
||||
const manifest = JSON.parse(
|
||||
document.getElementById('extension-surface').dataset.manifest
|
||||
);
|
||||
|
||||
// Full access to platform primitives + components
|
||||
const chatPane = window.ChatPane?.create(mount, { role: 'assist' });
|
||||
|
||||
// Build custom UI
|
||||
mount.innerHTML = '<h1>My Dashboard</h1>';
|
||||
})();
|
||||
```
|
||||
|
||||
**Admin UI additions:**
|
||||
|
||||
The admin surfaces section (already exists) gains:
|
||||
- Upload button (already wired to `InstallSurface`)
|
||||
- Enable/disable toggles (already wired)
|
||||
- Uninstall button (already wired to `DeleteSurface`)
|
||||
- Route display showing the `/s/:slug` URL
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] `surface-extension` template (HTML + CSS + scripts blocks)
|
||||
- [ ] `base.html` conditional chain extended with `{{if .Manifest}}` fallthrough
|
||||
- [ ] `loadExtensionSurfaces()` in page engine
|
||||
- [ ] nginx location block for `/surfaces/` static assets
|
||||
- [ ] Frontend nav renders extension surfaces from `ListEnabledSurfaces`
|
||||
- [ ] Admin surfaces section shows upload/enable/disable/uninstall
|
||||
- [ ] Sample `.surface` archive (hello-world dashboard) for testing
|
||||
- [ ] CSP nonce propagation for extension scripts
|
||||
|
||||
|
||||
### Phase 2: Workflow Engine Polish
|
||||
|
||||
Eight items deferred from v0.26.0. All backend infrastructure exists;
|
||||
these are wiring, UI, and enforcement.
|
||||
|
||||
**Channel header stage indicator + advance/reject controls**
|
||||
- Workflow channels show: stage name, step N of M, assignment info
|
||||
- Advance / Reject buttons (permission-gated by `workflow.manage`)
|
||||
- Renders in the chat header area (same slot as `ai_mode` context banner)
|
||||
|
||||
**Stage persona auto-switch in chat UI**
|
||||
- On stage transition, chat UI updates the active persona display
|
||||
- Model selector reflects the stage persona's bound model
|
||||
- System prompt injection already works (v0.26.4); this is FE-only
|
||||
|
||||
**Assignment notifications via WebSocket**
|
||||
- New workflow assignment → `workflow.assigned` WS event to team members
|
||||
- Claim confirmation → `workflow.claimed` WS event to claimer
|
||||
- Uses existing notification infrastructure (v0.20.0)
|
||||
|
||||
**Round-robin auto-assignment**
|
||||
- Per-stage `transition_rules.auto_assign: "round_robin"` in `workflow_stages`
|
||||
- On stage entry: query team members, pick least-recently-assigned
|
||||
- `workflow_assignments.assigned_to` populated automatically
|
||||
- Falls back to unassigned pool if round-robin fails
|
||||
|
||||
**`on_complete` workflow chaining**
|
||||
- Column exists on `workflows` table (nullable JSONB, v0.26.1)
|
||||
- Schema: `{"action": "start_workflow", "target_slug": "...", "data_map": {...}}`
|
||||
- On workflow completion: if `on_complete` is non-null, start target workflow
|
||||
with mapped `stage_data` from completed instance
|
||||
- Reuses existing `StartInstance()` path
|
||||
|
||||
**Workflow retention enforcement**
|
||||
- Column exists on `workflows` table (`retention` JSONB, v0.26.1)
|
||||
- Background sweep (extend staleness goroutine): completed instances
|
||||
older than `retention.delete_after_days` → hard delete
|
||||
- `retention.mode = "archive"` → set `workflow_status = 'archived'`,
|
||||
`ai_mode = 'off'` (already implemented for channel archive)
|
||||
|
||||
**Drag-and-drop stage reorder in builder**
|
||||
- Backend `PATCH /api/v1/workflows/:id/stages/reorder` already exists
|
||||
- Frontend: HTML5 drag events on stage list items in workflow builder
|
||||
- Same DnD pattern as channel/folder reorder (v0.23.1)
|
||||
|
||||
**Team-scoped workflow management UI**
|
||||
- Team admin settings surface gains "Workflows" section
|
||||
- Lists workflows owned by the team, quick-edit name/description
|
||||
- Links to full builder for stage editing
|
||||
- Mirrors the admin-level builder but scoped to `team_id`
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] Channel header workflow stage indicator component
|
||||
- [ ] Advance/reject buttons with permission gate
|
||||
- [ ] Stage persona auto-switch in chat UI
|
||||
- [ ] `workflow.assigned` / `workflow.claimed` WS events
|
||||
- [ ] Round-robin auto-assignment in stage transition handler
|
||||
- [ ] `on_complete` chaining trigger in workflow completion path
|
||||
- [ ] Retention enforcement in staleness sweep goroutine
|
||||
- [ ] Drag-and-drop stage reorder in workflow builder UI
|
||||
- [ ] Team settings → Workflows section
|
||||
|
||||
|
||||
### Phase 3: Workspace + Editor Debt
|
||||
|
||||
Items deferred from v0.21.x that are low-hanging fruit.
|
||||
|
||||
- [ ] `.gitignore` respect in workspace indexing (`workspace/indexer.go` — skip paths matching patterns from `.gitignore` in workspace root)
|
||||
- [ ] Workspace settings UI: git config section in channel/project settings (remote URL, branch, auto-commit toggle)
|
||||
- [ ] Pane state persistence per-user/per-project (save pane layout to `user_settings` JSONB, restore on surface load)
|
||||
- [ ] Editor drag-drop file reorder (workspace file tree — reorder via DnD, persist order in workspace metadata)
|
||||
|
||||
**Explicitly deferred (not v0.27.0):**
|
||||
- User settings git credentials management UI — needs vault-encrypted git credential store (v0.28.0 candidate)
|
||||
- Integration tests: clone/commit/push/pull — needs git binary in CI container
|
||||
- Article-specific AI tools — needs article surface rebuild
|
||||
- Mobile hamburger nav — needs responsive audit pass
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] `.gitignore` filter in `workspace/indexer.go`
|
||||
- [ ] Git config section in workspace/project settings UI
|
||||
- [ ] Pane layout persistence in `user_settings`
|
||||
- [ ] Editor file tree DnD reorder
|
||||
|
||||
|
||||
### ROADMAP Cleanup
|
||||
|
||||
Update the ROADMAP itself:
|
||||
- [ ] Cross off tool grant enforcement (3 locations)
|
||||
- [ ] Move "Deferred to v0.27.0" items under their proper v0.27.0 section
|
||||
- [ ] Remove duplicate entries (extension surface routes listed 4× across the file)
|
||||
- [ ] Add v0.27.0 section with phase structure
|
||||
- [ ] Update dependency graph
|
||||
|
||||
---
|
||||
|
||||
## v0.27.1 — Tasks Foundation: Service Channels + Scheduler
|
||||
|
||||
The core primitive: a channel with no human participant, driven by
|
||||
a scheduler.
|
||||
|
||||
### Service Channels
|
||||
|
||||
`type: 'service'` — a new channel type. Service channels:
|
||||
- Have zero human participants (only persona participants)
|
||||
- Cannot be joined by users (read-only view for authorized users)
|
||||
- Are created by the scheduler or the `task_create` tool
|
||||
- Inherit the workflow engine's stage model (optional — simple tasks
|
||||
skip stages entirely)
|
||||
- Appear in a "Tasks" sidebar section (collapsed by default)
|
||||
|
||||
```sql
|
||||
-- No new migration needed — channel type CHECK already allows extension.
|
||||
-- Add 'service' to the channel type CHECK constraint:
|
||||
ALTER TABLE channels DROP CONSTRAINT IF EXISTS channels_type_check;
|
||||
ALTER TABLE channels ADD CONSTRAINT channels_type_check
|
||||
CHECK (type IN ('direct','dm','group','channel','workflow','service'));
|
||||
```
|
||||
|
||||
### Task Definitions
|
||||
|
||||
A task is a lightweight scheduled job that creates a service channel
|
||||
and runs a completion (or instantiates a workflow).
|
||||
|
||||
**`tasks` table:**
|
||||
```sql
|
||||
CREATE TABLE tasks (
|
||||
id TEXT PRIMARY KEY DEFAULT gen_random_uuid()::text,
|
||||
owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
team_id TEXT REFERENCES teams(id) ON DELETE SET NULL,
|
||||
name TEXT NOT NULL,
|
||||
description TEXT,
|
||||
scope TEXT NOT NULL DEFAULT 'personal'
|
||||
CHECK (scope IN ('personal', 'team', 'global')),
|
||||
|
||||
-- What to run
|
||||
task_type TEXT NOT NULL DEFAULT 'prompt'
|
||||
CHECK (task_type IN ('prompt', 'workflow')),
|
||||
persona_id TEXT REFERENCES personas(id) ON DELETE SET NULL,
|
||||
model_id TEXT, -- explicit model override (nullable)
|
||||
system_prompt TEXT, -- additional system prompt (prepended)
|
||||
user_prompt TEXT, -- the message to send
|
||||
workflow_id TEXT REFERENCES workflows(id) ON DELETE SET NULL,
|
||||
tool_grants JSONB, -- explicit tool allowlist (nullable = inherit all)
|
||||
|
||||
-- Schedule
|
||||
schedule TEXT NOT NULL, -- cron expression (5-field)
|
||||
timezone TEXT NOT NULL DEFAULT 'UTC',
|
||||
is_active BOOLEAN NOT NULL DEFAULT true,
|
||||
|
||||
-- Execution policy
|
||||
max_tokens INTEGER NOT NULL DEFAULT 4096,
|
||||
max_tool_calls INTEGER NOT NULL DEFAULT 10,
|
||||
max_wall_clock INTEGER NOT NULL DEFAULT 300, -- seconds
|
||||
output_mode TEXT NOT NULL DEFAULT 'channel'
|
||||
CHECK (output_mode IN ('channel', 'note', 'webhook')),
|
||||
output_channel_id TEXT REFERENCES channels(id) ON DELETE SET NULL,
|
||||
webhook_url TEXT,
|
||||
|
||||
-- Provider routing
|
||||
provider_config_id TEXT REFERENCES provider_configs(id) ON DELETE SET NULL,
|
||||
|
||||
-- Notifications
|
||||
notify_on_complete BOOLEAN NOT NULL DEFAULT false,
|
||||
notify_on_failure BOOLEAN NOT NULL DEFAULT true,
|
||||
|
||||
-- Bookkeeping
|
||||
last_run_at TIMESTAMPTZ,
|
||||
next_run_at TIMESTAMPTZ,
|
||||
run_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_tasks_next_run ON tasks (next_run_at) WHERE is_active = true;
|
||||
CREATE INDEX idx_tasks_owner ON tasks (owner_id);
|
||||
```
|
||||
|
||||
**Two task types:**
|
||||
|
||||
1. **Prompt task** (`task_type = 'prompt'`): Create a service channel,
|
||||
send `user_prompt` with `system_prompt` prepended, collect response.
|
||||
Simple, single-turn. Good for: news digest, stock screener, daily
|
||||
standup prep, report generation.
|
||||
|
||||
2. **Workflow task** (`task_type = 'workflow'`): Instantiate a workflow
|
||||
in a service channel. Multi-stage, tool-using, potentially long-running.
|
||||
Good for: data pipeline, automated review, research agent.
|
||||
|
||||
### Scheduler
|
||||
|
||||
Background goroutine (same pattern as compaction scanner, staleness sweep):
|
||||
|
||||
```go
|
||||
type TaskScheduler struct {
|
||||
stores store.Stores
|
||||
interval time.Duration // poll interval: 30s
|
||||
stop chan struct{}
|
||||
}
|
||||
|
||||
func (s *TaskScheduler) Run() {
|
||||
ticker := time.NewTicker(s.interval)
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
s.poll()
|
||||
case <-s.stop:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *TaskScheduler) poll() {
|
||||
// SELECT * FROM tasks WHERE is_active AND next_run_at <= now()
|
||||
// ORDER BY next_run_at LIMIT 10
|
||||
due, _ := s.stores.Tasks.ListDue(ctx)
|
||||
for _, task := range due {
|
||||
go s.execute(task)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Cron parsing: use `github.com/robfig/cron/v3` for 5-field expressions.
|
||||
Compute `next_run_at` after each execution.
|
||||
|
||||
### Provider Resolution for Tasks
|
||||
|
||||
Task execution needs a provider. Resolution order:
|
||||
|
||||
1. Explicit `provider_config_id` on task (user chose a specific provider)
|
||||
2. Personal BYOK provider (if `scope = 'personal'` and user has BYOK keys)
|
||||
3. Team provider (if `scope = 'team'`)
|
||||
4. Global provider (fallback)
|
||||
5. Routing policy (if task's model matches a routing policy)
|
||||
|
||||
This is the same resolution chain as completions, reusing the existing
|
||||
`resolveProvider()` path in `completion.go`.
|
||||
|
||||
### Migration
|
||||
|
||||
- 026_tasks.sql: `tasks` table, service channel type extension
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] `tasks` table + store interface + PG/SQLite implementations
|
||||
- [ ] `type: 'service'` channel type
|
||||
- [ ] `TaskScheduler` background goroutine
|
||||
- [ ] Cron expression parsing + `next_run_at` computation
|
||||
- [ ] Task execution path (create service channel, run completion)
|
||||
- [ ] Provider resolution for task context
|
||||
- [ ] Tasks sidebar section (read-only view of service channels)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## v0.27.2 — Task Execution: Budgets + Admin Controls
|
||||
|
||||
### Execution Budgets
|
||||
|
||||
Three limits enforced in the task execution path:
|
||||
|
||||
1. **`max_tokens`** — total output tokens. Tracked via existing
|
||||
`usage_log` infrastructure. Execution halts mid-stream if exceeded.
|
||||
2. **`max_tool_calls`** — total tool invocations per run. Counter
|
||||
incremented in `streamWithToolLoop`. Execution halts on breach.
|
||||
3. **`max_wall_clock`** — seconds. `context.WithTimeout` on the
|
||||
execution goroutine. Hard kill on timeout.
|
||||
|
||||
Budget breach → task marked as `budget_exceeded` in run history,
|
||||
notification to owner.
|
||||
|
||||
### Task Run History
|
||||
|
||||
**`task_runs` table:**
|
||||
```sql
|
||||
CREATE TABLE task_runs (
|
||||
id TEXT PRIMARY KEY DEFAULT gen_random_uuid()::text,
|
||||
task_id TEXT NOT NULL REFERENCES tasks(id) ON DELETE CASCADE,
|
||||
channel_id TEXT REFERENCES channels(id) ON DELETE SET NULL,
|
||||
status TEXT NOT NULL DEFAULT 'running'
|
||||
CHECK (status IN ('running','completed','failed',
|
||||
'budget_exceeded','cancelled')),
|
||||
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
completed_at TIMESTAMPTZ,
|
||||
tokens_used INTEGER DEFAULT 0,
|
||||
tool_calls INTEGER DEFAULT 0,
|
||||
wall_clock INTEGER DEFAULT 0, -- seconds
|
||||
error TEXT
|
||||
);
|
||||
```
|
||||
|
||||
### Admin Controls
|
||||
|
||||
**Global config keys:**
|
||||
- `tasks.enabled` — master kill switch (default: true)
|
||||
- `tasks.allow_personal` — whether non-admin users can create personal
|
||||
tasks (default: false). Same pattern as `personas.allow_personal`.
|
||||
- `tasks.max_concurrent` — max simultaneous task executions (default: 5)
|
||||
- `tasks.default_budget.max_tokens` — default ceiling (overridable per task)
|
||||
- `tasks.default_budget.max_tool_calls` — default ceiling
|
||||
- `tasks.default_budget.max_wall_clock` — default ceiling (seconds)
|
||||
|
||||
**Admin panel — Tasks section:**
|
||||
- List all tasks (filterable by owner, team, scope, status)
|
||||
- View task run history with budget usage
|
||||
- Pause/resume individual tasks
|
||||
- Kill running executions
|
||||
- Edit default budgets
|
||||
- Toggle `tasks.allow_personal`
|
||||
|
||||
**Permission integration:**
|
||||
- New permission: `tasks.create` — required to create tasks
|
||||
- New permission: `tasks.admin` — required to manage others' tasks
|
||||
- `Everyone` group gets `tasks.create` if `tasks.allow_personal` is true
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] Execution budget enforcement (tokens, tool calls, wall clock)
|
||||
- [ ] `task_runs` table + store
|
||||
- [ ] Budget breach notification to task owner
|
||||
- [ ] Global config keys for task admin controls
|
||||
- [ ] Admin panel Tasks section (CRUD, history, kill switch)
|
||||
- [ ] `tasks.create` and `tasks.admin` permissions
|
||||
- [ ] `tasks.allow_personal` toggle (mirrors `personas.allow_personal`)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## v0.27.3 — Task Chaining: Webhooks + Workflow-to-Workflow
|
||||
|
||||
### Completion Webhooks
|
||||
|
||||
When a task (or workflow) completes, optionally POST to an external URL:
|
||||
|
||||
```go
|
||||
type WebhookPayload struct {
|
||||
TaskID string `json:"task_id"`
|
||||
TaskName string `json:"task_name"`
|
||||
Status string `json:"status"`
|
||||
ChannelID string `json:"channel_id"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
Output string `json:"output"` // last assistant message
|
||||
StageData any `json:"stage_data"` // workflow stage data (if applicable)
|
||||
}
|
||||
```
|
||||
|
||||
- Webhook URL on task or workflow (`webhook_url` column)
|
||||
- Retry: 3 attempts with exponential backoff (1s, 5s, 25s)
|
||||
- Timeout: 10s per attempt
|
||||
- HMAC signature header (`X-Switchboard-Signature`) using a per-task secret
|
||||
|
||||
### `task_create` Tool
|
||||
|
||||
AI-invocable tool that spawns sub-tasks:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "task_create",
|
||||
"description": "Create a new scheduled or one-shot task",
|
||||
"parameters": {
|
||||
"name": "string",
|
||||
"prompt": "string",
|
||||
"schedule": "string (cron or 'once')",
|
||||
"persona": "string (handle, optional)",
|
||||
"max_tokens": "integer (optional)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `RequireWorkflow` or `RequireTeam` predicate (not available in
|
||||
personal chats — prevents runaway task creation)
|
||||
- One-shot tasks: `schedule = 'once'`, `next_run_at = now()`
|
||||
- Created tasks inherit the parent's scope (team/global)
|
||||
- Depth limit: tasks cannot create tasks (no recursive spawning)
|
||||
|
||||
### Workflow-to-Workflow Chaining
|
||||
|
||||
Wires the `on_complete` column (v0.26.1) into the task system:
|
||||
|
||||
- On workflow completion with `on_complete` set: create a one-shot task
|
||||
that instantiates the target workflow with mapped data
|
||||
- Data mapping: `data_map` keys in `on_complete` JSONB map source
|
||||
`stage_data` fields to target workflow's initial `stage_data`
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] Webhook delivery with retry + HMAC signature
|
||||
- [ ] `task_create` tool with depth limit
|
||||
- [ ] `on_complete` chaining via task system
|
||||
- [ ] Webhook secret generation per task/workflow
|
||||
|
||||
|
||||
---
|
||||
|
||||
## v0.27.4 — Personal Tasks: BYOK Scheduling + User Task UI
|
||||
|
||||
The user-facing task experience. Users create personal tasks that run
|
||||
against their BYOK providers.
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Morning news digest:** Daily at 6am, prompt "Summarize top tech news",
|
||||
output to a personal service channel. Persona: "News Analyst" with
|
||||
`web_search` tool grant.
|
||||
- **Stock screener:** Weekdays at market open, prompt "Check my watchlist
|
||||
for unusual pre-market activity", BYOK OpenAI key for cost control.
|
||||
- **Weekly project summary:** Friday at 5pm, prompt "Summarize this
|
||||
week's activity across my projects", uses `conversation_search` +
|
||||
`workspace_search` tools.
|
||||
- **Daily standup prep:** Weekday mornings, reviews yesterday's notes
|
||||
and generates standup talking points.
|
||||
|
||||
### Settings Surface — Tasks Section
|
||||
|
||||
New section in user settings (same pattern as BYOK, Personas):
|
||||
|
||||
- Task list: name, schedule (human-readable), last run status, next run
|
||||
- Create task: name, persona picker, model picker, prompt editor,
|
||||
schedule builder (preset crons + custom), budget overrides
|
||||
- Task detail: run history, output channel link, edit, pause/delete
|
||||
- BYOK indicator: shows which provider the task will use
|
||||
|
||||
### Schedule Builder
|
||||
|
||||
Preset schedules + custom cron for power users:
|
||||
|
||||
| Preset | Cron |
|
||||
|--------|------|
|
||||
| Every morning (6am) | `0 6 * * *` |
|
||||
| Weekday mornings (8am) | `0 8 * * 1-5` |
|
||||
| Every hour | `0 * * * *` |
|
||||
| Weekly (Monday 9am) | `0 9 * * 1` |
|
||||
| Monthly (1st at midnight) | `0 0 1 * *` |
|
||||
| Custom... | user-entered 5-field cron |
|
||||
|
||||
Timezone selector defaults to browser timezone.
|
||||
|
||||
### BYOK Task Routing
|
||||
|
||||
Personal tasks prefer the user's BYOK provider:
|
||||
|
||||
1. If task has explicit `provider_config_id` → use it
|
||||
2. If user has a BYOK provider config for the task's model → use BYOK
|
||||
3. Fall through to team/global provider (if allowed by admin)
|
||||
|
||||
Admin can restrict personal tasks to BYOK-only via
|
||||
`tasks.personal_require_byok` config key (default: false). When true,
|
||||
personal tasks without a BYOK provider fail with a clear error instead
|
||||
of falling through to org providers.
|
||||
|
||||
### Output Modes
|
||||
|
||||
Three output destinations:
|
||||
|
||||
1. **Channel** (default): Output goes to a persistent service channel.
|
||||
User views it like a read-only chat. Channel accumulates run outputs
|
||||
over time (scrollable history).
|
||||
2. **Note**: Output saved as a channel-scoped note (good for structured
|
||||
data, reports). Note title includes timestamp.
|
||||
3. **Webhook**: Output POSTed to external URL (for integration with
|
||||
other tools — Slack, email, etc.).
|
||||
|
||||
### Task Sidebar Section
|
||||
|
||||
New sidebar section (below Channels, above Chats):
|
||||
|
||||
```
|
||||
▾ Tasks (collapsible)
|
||||
◷ Morning News Digest ✓ 6:02am
|
||||
◷ Stock Screener ✓ 9:30am
|
||||
◷ Weekly Summary ⏳ Fri 5pm
|
||||
+ New task
|
||||
```
|
||||
|
||||
Click opens the task's output channel. Status indicators: ✓ (last run
|
||||
succeeded), ✗ (failed), ⏳ (next run time), ▶ (running now).
|
||||
|
||||
**Deliverables:**
|
||||
- [ ] Settings → Tasks section (CRUD, schedule builder, budget config)
|
||||
- [ ] BYOK provider routing for personal tasks
|
||||
- [ ] `tasks.personal_require_byok` config key
|
||||
- [ ] Output modes: channel, note, webhook
|
||||
- [ ] Tasks sidebar section with status indicators
|
||||
- [ ] "Run Now" button for manual trigger
|
||||
- [ ] Task output channel read-only view
|
||||
|
||||
|
||||
---
|
||||
|
||||
## v0.28.0 — Platform Polish (TBD Pull-Forward)
|
||||
|
||||
Candidates pulled from the TBD section, prioritized by value and
|
||||
dependency readiness.
|
||||
|
||||
### Tier 1 — High value, dependencies met
|
||||
|
||||
**Virtual scroll for long conversations**
|
||||
Conversations with 500+ messages bog down the DOM. Virtual scroll renders
|
||||
only visible messages + buffer zone. Prerequisite for heavy task output
|
||||
channels.
|
||||
|
||||
**KB auto-injection (context-aware)**
|
||||
Top-K chunk prepend to system prompt, context budget aware, per-channel
|
||||
toggle. Useful for tasks that need domain knowledge without explicit
|
||||
`kb_search` tool calls. Latency budgeting required (embedding lookup
|
||||
adds ~200ms).
|
||||
|
||||
**Helm chart**
|
||||
Raw k8s manifests with `${VAR}` substitution → proper Helm chart.
|
||||
`values.yaml` for replicas, image tags, ingress, storage, secrets,
|
||||
feature flags. Subchart for dev/test Postgres. Target: `helm install
|
||||
switchboard ./chart`. Now makes sense because the feature set is
|
||||
stabilizing post-tasks.
|
||||
|
||||
**Per-provider model preferences**
|
||||
`user_model_settings` unique key is `(user_id, model_id)` — same model
|
||||
from different providers shares one visibility toggle. Needs
|
||||
`provider_config_id` dimension in DB constraint, store, API, and
|
||||
frontend `hiddenModels` keying. Natural fit now that BYOK tasks need
|
||||
per-provider model selection.
|
||||
|
||||
### Tier 2 — Medium value, some design work needed
|
||||
|
||||
**Memory compaction**
|
||||
Summarize old memories to save context tokens. Confidence decay: reduce
|
||||
confidence over time, prune low-confidence entries. Natural extension of
|
||||
the memory system (v0.18.0).
|
||||
|
||||
**`capability_match` routing policy**
|
||||
"Cheapest model with tool_calling" — a new routing policy type. Useful
|
||||
for tasks that need tool use but don't need the strongest model.
|
||||
|
||||
**New provider types via config file**
|
||||
OpenAI-compatible endpoints registrable via YAML config (no Go code).
|
||||
Enables users to point at local LLMs (Ollama, vLLM, llama.cpp) without
|
||||
a provider code change.
|
||||
|
||||
### Tier 3 — Future (v0.29+)
|
||||
|
||||
- Desktop app (Tauri) — large scope, independent track
|
||||
- Full PWA with offline — needs service worker rewrite
|
||||
- Plugin/extension marketplace — needs extension surface routes first (v0.27.0)
|
||||
- Starlark/sidecar extension tiers — needs extension surface routes first
|
||||
- Git credentials vault store — needs vault extension design
|
||||
- Cross-persona memory sharing — needs memory system redesign
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Design Decision: Task Permission Model
|
||||
|
||||
Tasks introduce a new resource type that intersects with multiple
|
||||
existing permission boundaries:
|
||||
|
||||
| Scope | Who can create | Provider used | Visibility |
|
||||
|-------|---------------|---------------|------------|
|
||||
| Personal | User (if `tasks.allow_personal`) | User's BYOK or team fallback | Owner only |
|
||||
| Team | User with `tasks.create` + team membership | Team provider | Team members |
|
||||
| Global | Admin | Global provider | All users (read) |
|
||||
|
||||
Personal tasks are the BYOK sweet spot. The admin toggle
|
||||
(`tasks.allow_personal`) mirrors the existing `personas.allow_personal`
|
||||
pattern — a single boolean in global config, surfaced in the admin
|
||||
settings panel.
|
||||
|
||||
**Why not just a permission?** The `tasks.create` permission already
|
||||
gates the ability. `tasks.allow_personal` is a _policy_ control —
|
||||
admins might allow task creation for team tasks but prohibit personal
|
||||
tasks to prevent uncontrolled BYOK spending. Two orthogonal knobs.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Design Decision: Task vs. Workflow Relationship
|
||||
|
||||
Tasks and workflows are related but distinct:
|
||||
|
||||
| | Task | Workflow |
|
||||
|---|------|---------|
|
||||
| Trigger | Cron schedule or `task_create` tool | Human click or task trigger |
|
||||
| Participants | Zero humans (service channel) | Humans + AI |
|
||||
| Complexity | Single prompt or workflow instantiation | Multi-stage, assignment queue |
|
||||
| Output | Channel, note, or webhook | Channel (interactive) |
|
||||
| Lifecycle | Recurring (cron) or one-shot | Single instance |
|
||||
|
||||
A **prompt task** is simpler than a workflow — it's a scheduled
|
||||
completion. A **workflow task** is a task that instantiates a workflow.
|
||||
This keeps the task system lean (scheduling + budgets + output routing)
|
||||
and the workflow engine rich (stages + assignment + personas).
|
||||
|
||||
The `task_type` column makes this explicit. Prompt tasks don't need
|
||||
workflows at all — they're a direct scheduler → completion path.
|
||||
Workflow tasks reuse the full workflow engine.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Resolved Decisions
|
||||
|
||||
1. **Task output retention.** Use channel-level retention from v0.26.0.
|
||||
No per-task retention policy needed — the channel's `retention` JSONB
|
||||
handles archive/delete semantics. Notes created by task output survive
|
||||
channel removal (notes are channel-scoped but not cascade-deleted).
|
||||
Storage quotas per user are a future concern (v0.29+ candidate) — not
|
||||
blocking for initial task support.
|
||||
|
||||
2. **Concurrent task execution.** Skip with a warning log. If
|
||||
`task_runs` has a row with `status = 'running'` for the task, the
|
||||
scheduler skips that tick. Log: `[scheduler] Skipping task %s — previous
|
||||
run still active`. No queue — if a task consistently overruns its
|
||||
schedule, the user needs to adjust the cron interval or budget.
|
||||
|
||||
3. **Task templates.** Ship 3–5 optional starter templates in v0.27.4.
|
||||
Presented as suggestions during task creation ("Start from a
|
||||
template..." option alongside blank task). Not auto-created — user
|
||||
must explicitly choose one. Templates are just pre-filled form values,
|
||||
not persistent DB records.
|
||||
|
||||
4. **Task notifications.** Opt-in per task. Two booleans on the `tasks`
|
||||
table: `notify_on_complete` (default false), `notify_on_failure`
|
||||
(default true). Uses existing notification infrastructure (v0.20.0
|
||||
bell + WebSocket push).
|
||||
498
docs/archive/DESIGN-CM6.md
Normal file
498
docs/archive/DESIGN-CM6.md
Normal file
@@ -0,0 +1,498 @@
|
||||
# DESIGN: CodeMirror 6 Integration
|
||||
|
||||
**Version:** v0.17.2
|
||||
**Status:** Complete
|
||||
**Scope:** Add CM6 as a vendored dependency, bundled at Docker build time via esbuild. Three integration surfaces.
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Three features converge on the same dependency:
|
||||
|
||||
1. **Chat input** — live markdown rendering (backtick → code block, bold, etc.)
|
||||
2. **Extension editor** — syntax-highlighted JavaScript/JSON editing in admin panel (replaces bare `<textarea>`)
|
||||
3. **Code editing surface** (roadmap) — future extension surface type for code review, snippets, etc.
|
||||
|
||||
CM6 is ESM-native and expects a bundler. The build step is contained in the Docker image build — the frontend continues to ship as static assets served by nginx. No runtime build tooling, no change to the developer experience for non-CM6 code.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Build Pipeline
|
||||
|
||||
New Docker stage between the existing `vendor` stage and the final `nginx` stage:
|
||||
|
||||
```
|
||||
Stage 1: vendor (existing — marked, purify, mermaid, katex)
|
||||
Stage 2: cm6-build (NEW — npm install + esbuild → single IIFE bundle)
|
||||
Stage 3: nginx (existing — copies src/ + vendor/ + cm6 bundle)
|
||||
```
|
||||
|
||||
The CM6 build stage:
|
||||
|
||||
```dockerfile
|
||||
FROM node:20-alpine AS cm6-build
|
||||
WORKDIR /build
|
||||
COPY src/editor/package.json src/editor/build.mjs ./
|
||||
RUN npm install --production=false
|
||||
RUN node build.mjs
|
||||
# Output: /build/dist/codemirror.bundle.js (~180-250KB minified)
|
||||
# /build/dist/codemirror.bundle.css (~5-10KB)
|
||||
```
|
||||
|
||||
Final stage addition:
|
||||
|
||||
```dockerfile
|
||||
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
|
||||
```
|
||||
|
||||
### Bundle Structure
|
||||
|
||||
A single esbuild entrypoint (`src/editor/index.mjs`) that imports CM6 packages and exposes factory functions on `window.CM`:
|
||||
|
||||
```javascript
|
||||
// src/editor/index.mjs — esbuild entrypoint
|
||||
import { EditorView, keymap, placeholder, ViewPlugin, ... } from '@codemirror/view';
|
||||
import { EditorState, ... } from '@codemirror/state';
|
||||
import { markdown, markdownLanguage } from '@codemirror/lang-markdown';
|
||||
import { javascript } from '@codemirror/lang-javascript';
|
||||
import { json } from '@codemirror/lang-json';
|
||||
import { sql } from '@codemirror/lang-sql';
|
||||
import { html } from '@codemirror/lang-html';
|
||||
import { css } from '@codemirror/lang-css';
|
||||
import { yaml } from '@codemirror/lang-yaml';
|
||||
import { go } from '@codemirror/lang-go';
|
||||
import { python } from '@codemirror/lang-python';
|
||||
import { rust } from '@codemirror/lang-rust';
|
||||
import { oneDark } from '@codemirror/theme-one-dark';
|
||||
import { defaultKeymap, indentWithTab } from '@codemirror/commands';
|
||||
import { bracketMatching, ... } from '@codemirror/language';
|
||||
import { highlightSelectionMatches, searchKeymap } from '@codemirror/search';
|
||||
import { autocompletion } from '@codemirror/autocomplete';
|
||||
import { vim } from '@replit/codemirror-vim';
|
||||
import { emacs } from '@replit/codemirror-emacs';
|
||||
|
||||
// Expose on window for script-tag consumption
|
||||
window.CM = {
|
||||
EditorView,
|
||||
EditorState,
|
||||
|
||||
// Factory: chat input (markdown mode, minimal chrome)
|
||||
chatInput(target, opts = {}) { ... },
|
||||
|
||||
// Factory: code editor (full features, language auto-detect)
|
||||
codeEditor(target, opts = {}) { ... },
|
||||
|
||||
// Supported languages for code editor
|
||||
languages: { markdown, javascript, json, sql, html, css, yaml, go, python, rust },
|
||||
|
||||
// Keybinding modes (applied via user preference)
|
||||
keybindings: { vim, emacs },
|
||||
};
|
||||
```
|
||||
|
||||
### esbuild Config
|
||||
|
||||
```javascript
|
||||
// src/editor/build.mjs
|
||||
import { build } from 'esbuild';
|
||||
|
||||
await build({
|
||||
entryPoints: ['index.mjs'],
|
||||
bundle: true,
|
||||
minify: true,
|
||||
format: 'iife',
|
||||
target: ['es2020'],
|
||||
outfile: 'dist/codemirror.bundle.js',
|
||||
// CSS extracted automatically by esbuild
|
||||
});
|
||||
```
|
||||
|
||||
### package.json (editor only)
|
||||
|
||||
```json
|
||||
{
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"devDependencies": {
|
||||
"esbuild": "^0.25.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@codemirror/autocomplete": "^6",
|
||||
"@codemirror/commands": "^6",
|
||||
"@codemirror/lang-css": "^6",
|
||||
"@codemirror/lang-go": "^6",
|
||||
"@codemirror/lang-html": "^6",
|
||||
"@codemirror/lang-javascript": "^6",
|
||||
"@codemirror/lang-json": "^6",
|
||||
"@codemirror/lang-markdown": "^6",
|
||||
"@codemirror/lang-sql": "^6",
|
||||
"@codemirror/lang-python": "^6",
|
||||
"@codemirror/lang-rust": "^6",
|
||||
"@codemirror/lang-yaml": "^6",
|
||||
"@codemirror/language": "^6",
|
||||
"@codemirror/search": "^6",
|
||||
"@codemirror/state": "^6",
|
||||
"@codemirror/theme-one-dark": "^6",
|
||||
"@codemirror/view": "^6",
|
||||
"@replit/codemirror-vim": "^6",
|
||||
"@replit/codemirror-emacs": "^6"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Surfaces
|
||||
|
||||
### 1. Chat Input — Markdown Mode
|
||||
|
||||
Replace `<textarea id="messageInput">` with a CM6 instance configured for chat composition.
|
||||
|
||||
**Behavior:**
|
||||
|
||||
- Markdown syntax highlighting as you type (fenced code blocks, bold, italic, links, etc.)
|
||||
- Shift+Enter inserts newline (existing behavior preserved)
|
||||
- Enter sends message (existing behavior preserved)
|
||||
- Auto-growing height (CM6 handles this natively via `EditorView.contentAttributes`)
|
||||
- Placeholder text: "Send a message..."
|
||||
- No line numbers, no gutter — minimal chrome
|
||||
- Tab inserts real tab (indentWithTab) inside code blocks; normal behavior outside
|
||||
|
||||
**Factory:**
|
||||
|
||||
```javascript
|
||||
CM.chatInput(document.getElementById('messageInputContainer'), {
|
||||
placeholder: 'Send a message...',
|
||||
onSubmit: (text) => sendMessage(text), // Enter key
|
||||
onChange: (text) => updateInputTokens(text), // live token count
|
||||
darkMode: true,
|
||||
});
|
||||
```
|
||||
|
||||
**Migration path:**
|
||||
|
||||
The existing code reads `document.getElementById('messageInput').value` in many places. The factory returns an object with a `.getValue()` / `.setValue()` / `.focus()` API that matches the textarea interface. A thin shim keeps all existing callsites working:
|
||||
|
||||
```javascript
|
||||
// After CM.chatInput() creates the editor:
|
||||
const editor = CM.chatInput(...);
|
||||
|
||||
// Shim: existing code that reads .value still works
|
||||
Object.defineProperty(document.getElementById('messageInput'), 'value', {
|
||||
get: () => editor.getValue(),
|
||||
set: (v) => editor.setValue(v),
|
||||
});
|
||||
```
|
||||
|
||||
Alternatively (cleaner): introduce `ChatInput.getValue()` / `ChatInput.setValue()` / `ChatInput.focus()` and update callsites. There aren't many — `chat.js`, `attachments.js`, `tokens.js`, and a few event handlers.
|
||||
|
||||
### 2. Extension Editor — JavaScript/JSON Mode
|
||||
|
||||
Replace the two `<textarea>` elements in `editAdminExtension()` (`admin-handlers.js` line 706-710) with CM6 instances.
|
||||
|
||||
**Behavior:**
|
||||
|
||||
- Manifest textarea → JSON mode with bracket matching, auto-indent
|
||||
- Script textarea → JavaScript mode with bracket matching, auto-indent
|
||||
- Line numbers enabled
|
||||
- Search/replace (Ctrl+F) — currently impossible in a textarea
|
||||
- Tab key inserts proper indent (replaces the manual keydown handler at line 726-736)
|
||||
- Themed to match the app's dark/light mode
|
||||
|
||||
**Factory:**
|
||||
|
||||
```javascript
|
||||
CM.codeEditor(document.getElementById('extEdit-manifest-container'), {
|
||||
language: 'json',
|
||||
value: manifestJSON,
|
||||
lineNumbers: true,
|
||||
darkMode: isDark,
|
||||
});
|
||||
|
||||
CM.codeEditor(document.getElementById('extEdit-script-container'), {
|
||||
language: 'javascript',
|
||||
value: script,
|
||||
lineNumbers: true,
|
||||
darkMode: isDark,
|
||||
keymap: prefs.editorKeymap || 'standard', // 'standard' | 'vim' | 'emacs'
|
||||
});
|
||||
```
|
||||
|
||||
**Save integration:** `saveAdminExtension()` currently reads `.value` from the textareas. The CM6 instances expose `.getValue()` — update the save function to call that instead.
|
||||
|
||||
### 3. Future Code Surface (Roadmap)
|
||||
|
||||
The `CM.codeEditor()` factory with language auto-detection covers this. No additional work needed now — just document that it's available for the extensions surface type when that ships.
|
||||
|
||||
---
|
||||
|
||||
## Loading Strategy
|
||||
|
||||
The CM6 bundle is **not** in `SHELL_FILES` (service worker precache list). It loads on demand:
|
||||
|
||||
```html
|
||||
<!-- index.html: load after core app scripts, before app.js init -->
|
||||
<script src="vendor/codemirror/codemirror.bundle.js?v=%%APP_VERSION%%"
|
||||
onerror="console.warn('CM6 not available — falling back to textarea')"></script>
|
||||
<link rel="stylesheet" href="vendor/codemirror/codemirror.bundle.css?v=%%APP_VERSION%%"
|
||||
onerror="this.remove()">
|
||||
```
|
||||
|
||||
**Graceful degradation:** If the bundle fails to load (disconnected environment without vendored copy, build issue, etc.), `window.CM` is undefined and all integration points fall back to plain `<textarea>`. Each factory callsite checks:
|
||||
|
||||
```javascript
|
||||
if (window.CM) {
|
||||
CM.chatInput(container, opts);
|
||||
} else {
|
||||
// existing textarea behavior — unchanged
|
||||
}
|
||||
```
|
||||
|
||||
This means CM6 is a progressive enhancement, not a hard dependency.
|
||||
|
||||
---
|
||||
|
||||
## SW Cache Considerations
|
||||
|
||||
Add `vendor/codemirror/` to the SW exclusion list (same fix as the extensions path discussed earlier):
|
||||
|
||||
```javascript
|
||||
if (url.pathname.includes('/api/') ||
|
||||
url.pathname.includes('/ws') ||
|
||||
url.pathname.includes('/branding/') ||
|
||||
url.pathname.includes('/extensions/') ||
|
||||
url.pathname.includes('/vendor/codemirror/') ||
|
||||
event.request.method !== 'GET') {
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
Or, more broadly, exclude all of `/vendor/` if you want vendor updates to always bypass the SW cache. The core vendors (marked, purify) are small enough that network-first is fine.
|
||||
|
||||
Alternatively, since the bundle is versioned via `?v=%%APP_VERSION%%` and only changes on new builds, it could safely be in `SHELL_FILES` for precaching — unlike extensions which are user-editable at runtime. Either approach works.
|
||||
|
||||
---
|
||||
|
||||
## Theme Integration
|
||||
|
||||
CM6's `oneDark` theme for dark mode. For light mode, the CM6 default is clean and neutral.
|
||||
|
||||
Both need CSS variable overrides to match the app's existing `--bg`, `--bg-2`, `--border`, `--text`, `--accent` palette:
|
||||
|
||||
```javascript
|
||||
const switchboardTheme = EditorView.theme({
|
||||
'&': {
|
||||
backgroundColor: 'var(--bg)',
|
||||
color: 'var(--text)',
|
||||
fontSize: '14px',
|
||||
},
|
||||
'.cm-content': {
|
||||
fontFamily: 'var(--mono)',
|
||||
caretColor: 'var(--accent)',
|
||||
},
|
||||
'.cm-cursor': {
|
||||
borderLeftColor: 'var(--accent)',
|
||||
},
|
||||
'&.cm-focused .cm-selectionBackground, .cm-selectionBackground': {
|
||||
backgroundColor: 'var(--accent-muted, rgba(99,102,241,0.2))',
|
||||
},
|
||||
'.cm-gutters': {
|
||||
backgroundColor: 'var(--bg-2)',
|
||||
color: 'var(--text-3)',
|
||||
borderRight: '1px solid var(--border)',
|
||||
},
|
||||
}, { dark: document.body.classList.contains('dark-theme') });
|
||||
```
|
||||
|
||||
This goes into the bundle so it's applied automatically.
|
||||
|
||||
---
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
src/
|
||||
editor/
|
||||
package.json # CM6 deps + esbuild
|
||||
build.mjs # esbuild script
|
||||
index.mjs # bundle entrypoint, exports window.CM
|
||||
theme.mjs # switchboard theme overrides
|
||||
chat-input.mjs # chatInput() factory
|
||||
code-editor.mjs # codeEditor() factory
|
||||
js/
|
||||
... # existing app code (unchanged)
|
||||
vendor/
|
||||
marked.min.js # existing
|
||||
purify.min.js # existing
|
||||
codemirror/ # gitignored — built by Docker
|
||||
codemirror.bundle.js
|
||||
codemirror.bundle.css
|
||||
```
|
||||
|
||||
The `src/editor/` directory is self-contained. It has its own `package.json` and `node_modules` (inside Docker only). No npm dependencies bleed into the main `src/js/` code. The build output lands in `vendor/codemirror/` which is gitignored like the other Docker-built vendors.
|
||||
|
||||
---
|
||||
|
||||
## Dockerfile Changes
|
||||
|
||||
```dockerfile
|
||||
# Stage 1: vendor libs (existing)
|
||||
FROM node:20-alpine AS vendor
|
||||
# ... existing marked, purify, mermaid, katex extraction ...
|
||||
|
||||
# Stage 2: CM6 bundle (NEW)
|
||||
FROM node:20-alpine AS cm6-build
|
||||
WORKDIR /build
|
||||
COPY src/editor/ ./
|
||||
RUN npm ci
|
||||
RUN node build.mjs
|
||||
|
||||
# Stage 3: nginx (existing, with additions)
|
||||
FROM nginx:1-alpine
|
||||
# ... existing setup ...
|
||||
COPY src/ /usr/share/nginx/html/
|
||||
COPY --from=vendor /vendor/ /usr/share/nginx/html/vendor/
|
||||
COPY --from=cm6-build /build/dist/ /usr/share/nginx/html/vendor/codemirror/
|
||||
# ... rest unchanged ...
|
||||
```
|
||||
|
||||
The `vendor` and `cm6-build` stages run in parallel (Docker BuildKit), so build time impact is minimal.
|
||||
|
||||
---
|
||||
|
||||
## Migration Checklist
|
||||
|
||||
### Phase 1: Bundle + Extension Editor
|
||||
- [ ] Create `src/editor/` with package.json, build.mjs, index.mjs
|
||||
- [ ] Implement `CM.codeEditor()` factory
|
||||
- [ ] Update Dockerfile.frontend with cm6-build stage
|
||||
- [ ] Replace extension editor textareas in `admin-handlers.js`
|
||||
- [ ] Remove manual Tab key handler (CM6 handles it)
|
||||
- [ ] Verify save/load round-trip with CM6 `.getValue()`
|
||||
- [ ] Add graceful fallback if `window.CM` undefined
|
||||
|
||||
### Phase 2: Chat Input
|
||||
- [ ] Implement `CM.chatInput()` factory with markdown mode
|
||||
- [ ] Wire Enter=send, Shift+Enter=newline keybindings
|
||||
- [ ] Integrate with `updateInputTokens()` via onChange callback
|
||||
- [ ] Update `chat.js`, `attachments.js`, `tokens.js` to use CM API
|
||||
- [ ] Auto-grow height behavior matching current textarea
|
||||
- [ ] Test paste handling (plain text, code, attachments)
|
||||
- [ ] Test mobile/touch input
|
||||
|
||||
### Phase 3: Polish
|
||||
- [ ] Theme integration with CSS variables
|
||||
- [ ] Dark/light mode switching (listen for theme toggle)
|
||||
- [ ] Editor keymap preference UI (Standard / Vim / Emacs) in appearance settings
|
||||
- [ ] Vim/Emacs modes on code editor + extension editor only (not chat input)
|
||||
- [ ] Add to `SHELL_FILES` in sw.js (or exclude from SW cache)
|
||||
- [ ] Update `debug.js` snapshot to include CM6 version
|
||||
- [ ] Documentation in ARCHITECTURE.md
|
||||
|
||||
---
|
||||
|
||||
## Bundle Size Estimate
|
||||
|
||||
Based on CM6 package sizes (minified + tree-shaken by esbuild):
|
||||
|
||||
| Package | Approx Size |
|
||||
|---------|-------------|
|
||||
| @codemirror/view + state + commands | ~90KB |
|
||||
| @codemirror/language + highlight | ~30KB |
|
||||
| lang-markdown + lang-javascript + lang-json | ~40KB |
|
||||
| lang-go + lang-sql + lang-html + lang-css + lang-yaml | ~50KB |
|
||||
| lang-python + lang-rust | ~20KB |
|
||||
| theme-one-dark | ~5KB |
|
||||
| search + autocomplete | ~20KB |
|
||||
| @replit/codemirror-vim + emacs | ~40KB |
|
||||
| **Total (minified)** | **~295KB** |
|
||||
| **Gzipped** | **~90KB** |
|
||||
|
||||
For comparison: mermaid.min.js is ~1.2MB. This is modest.
|
||||
|
||||
---
|
||||
|
||||
## Resolved Decisions
|
||||
|
||||
1. **Language modes** — Ship Python, Rust, and YAML upfront alongside Go, JS, JSON, SQL, HTML, CSS, and Markdown. Covers the team's stack. Additional modes are a one-line import + rebuild.
|
||||
|
||||
2. **Local dev build script** — Provide a `scripts/build-editor.sh` that runs the esbuild step standalone, shared by both Dockerfile.frontend and the unified Dockerfile. Developers can also run it locally to get the CM6 bundle without a full Docker build.
|
||||
|
||||
3. **Vim/Emacs keybindings** — Ship both. Exposed as a user preference toggle (default: standard). Applied via `CM.keybindings.vim` / `CM.keybindings.emacs` as CM6 extensions at editor creation time.
|
||||
|
||||
---
|
||||
|
||||
## Build Script
|
||||
|
||||
Shared script used by both Dockerfiles and local dev:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
# scripts/build-editor.sh — Build CM6 bundle
|
||||
# Usage: ./scripts/build-editor.sh [output-dir]
|
||||
#
|
||||
# Defaults to src/vendor/codemirror/ for local dev.
|
||||
# Dockerfiles pass /build/dist/ as the output dir.
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
EDITOR_DIR="${SCRIPT_DIR}/../src/editor"
|
||||
OUTPUT_DIR="${1:-${SCRIPT_DIR}/../src/vendor/codemirror}"
|
||||
|
||||
cd "${EDITOR_DIR}"
|
||||
|
||||
# Install if needed (CI/Docker will already have node_modules)
|
||||
[ -d node_modules ] || npm ci
|
||||
|
||||
mkdir -p "${OUTPUT_DIR}"
|
||||
node build.mjs --outdir="${OUTPUT_DIR}"
|
||||
|
||||
echo "✅ CM6 bundle → ${OUTPUT_DIR}"
|
||||
```
|
||||
|
||||
Dockerfile usage:
|
||||
|
||||
```dockerfile
|
||||
FROM node:20-alpine AS cm6-build
|
||||
WORKDIR /build
|
||||
COPY src/editor/ /build/src/editor/
|
||||
COPY scripts/build-editor.sh /build/scripts/build-editor.sh
|
||||
RUN cd /build/src/editor && npm ci
|
||||
RUN sh /build/scripts/build-editor.sh /build/dist
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Keybinding Preference
|
||||
|
||||
Add to user appearance preferences (alongside dark/light theme toggle):
|
||||
|
||||
```
|
||||
Editor keybindings: [Standard ▾] [Vim] [Emacs]
|
||||
```
|
||||
|
||||
Stored in `localStorage` under `cs-appearance`:
|
||||
|
||||
```javascript
|
||||
const prefs = JSON.parse(localStorage.getItem('cs-appearance') || '{}');
|
||||
// prefs.editorKeymap = 'standard' | 'vim' | 'emacs'
|
||||
```
|
||||
|
||||
Applied at editor creation time:
|
||||
|
||||
```javascript
|
||||
const keymapExt = [];
|
||||
if (prefs.editorKeymap === 'vim') keymapExt.push(CM.keybindings.vim());
|
||||
else if (prefs.editorKeymap === 'emacs') keymapExt.push(CM.keybindings.emacs());
|
||||
|
||||
CM.codeEditor(target, {
|
||||
language: 'javascript',
|
||||
extensions: keymapExt,
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
**Note:** Vim/Emacs modes apply only to the code editor and extension editor surfaces. The chat input always uses standard keybindings to avoid confusing Enter-to-send behavior with modal editing.
|
||||
186
docs/archive/v0171-sqlite-backend.md
Normal file
186
docs/archive/v0171-sqlite-backend.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# v0.17.1 — SQLite Backend
|
||||
|
||||
## Overview
|
||||
|
||||
Adds SQLite as an alternative database backend alongside PostgreSQL. Enables
|
||||
single-binary deployments, air-gapped environments, edge nodes, and developer
|
||||
laptops without requiring a PostgreSQL instance.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
DB_DRIVER env var
|
||||
│
|
||||
├─ "postgres" (default) → database.DialectPostgres → postgres.NewStores()
|
||||
└─ "sqlite" → database.DialectSQLite → sqlite.NewStores()
|
||||
|
||||
Both implement the same 19 store.* interfaces unchanged.
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Parallel `store/sqlite/` package | Clean separation, no runtime branching in SQL |
|
||||
| `?` placeholders (not `$N`) | SQLite native, no conversion overhead |
|
||||
| JSON text arrays (`'["a","b"]'`) | Replace `TEXT[]`/`UUID[]` + `pq.Array` |
|
||||
| `json_each()` for array membership | Replace `= ANY(array_col)` |
|
||||
| App-side `store.NewID()` UUIDs | Replace `DEFAULT gen_random_uuid()` |
|
||||
| `datetime('now')` | Replace `NOW()` / `CURRENT_TIMESTAMP` |
|
||||
| `LIKE` fallback for search | Replace `tsvector` / `ts_rank` |
|
||||
| Feature-gated vector search | pgvector → returns error hint on SQLite |
|
||||
| `excluded.col` in ON CONFLICT | Replace Postgres `$N` reuse in SET clause |
|
||||
| WAL mode + single writer | Optimal SQLite concurrency for web apps |
|
||||
|
||||
## Files Delivered
|
||||
|
||||
### Infrastructure
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server/config/config.go` | Added `DBDriver` field + `DB_DRIVER` env var |
|
||||
| `server/database/dialect.go` | `Dialect` type, `IsPostgres()`, `IsSQLite()` |
|
||||
| `server/database/database.go` | Dual-driver `Connect()`: postgres + sqlite |
|
||||
| `server/database/migrate.go` | Dialect-aware migration runner |
|
||||
| `server/database/migrations/sqlite/001_v017_schema.sql` | Full SQLite schema |
|
||||
| `server/store/id.go` | `store.NewID()` — app-side UUID generation |
|
||||
|
||||
### Store Layer (19 stores)
|
||||
| File | Lines | Key Transforms |
|
||||
|------|-------|---------------|
|
||||
| `store/sqlite/helpers.go` | Query builders with `?`, JSON helpers |
|
||||
| `store/sqlite/stores.go` | `NewStores()` constructor |
|
||||
| `store/sqlite/user.go` | `store.NewID()`, `datetime('now')` |
|
||||
| `store/sqlite/provider.go` | JSONB→TEXT, `store.NewID()` |
|
||||
| `store/sqlite/catalog.go` | JSONB→TEXT scanning |
|
||||
| `store/sqlite/persona.go` | `json_each()` for group grants |
|
||||
| `store/sqlite/policy.go` | `excluded.col` in upsert |
|
||||
| `store/sqlite/user_settings.go` | COALESCE with `excluded.col` |
|
||||
| `store/sqlite/channel.go` | Tags as JSON text array |
|
||||
| `store/sqlite/message.go` | `store.NewID()` |
|
||||
| `store/sqlite/note.go` | LIKE search fallback, JSON tags |
|
||||
| `store/sqlite/usage.go` | Dynamic `?` filters |
|
||||
| `store/sqlite/pricing.go` | `excluded.col` upserts |
|
||||
| `store/sqlite/extension.go` | `store.NewID()` |
|
||||
| `store/sqlite/attachment.go` | `store.NewID()` |
|
||||
| `store/sqlite/knowledge_bases.go` | `IN(?)` replaces `ANY()`, vector search gated |
|
||||
| `store/sqlite/groups.go` | `store.NewID()` |
|
||||
| `store/sqlite/resource_grants.go` | `json_each()` for array membership |
|
||||
| `store/sqlite/team.go` | `store.NewID()`, `excluded.col` |
|
||||
| `store/sqlite/audit.go` | `store.NewID()` |
|
||||
| `store/sqlite/global_config.go` | `excluded.col` upsert |
|
||||
|
||||
### Integration
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server/MAIN_GO_PATCH.md` | Shows exact main.go changes needed |
|
||||
|
||||
## Pattern Conversion Reference
|
||||
|
||||
### Placeholders
|
||||
```sql
|
||||
-- Postgres: WHERE id = $1 AND name = $2
|
||||
-- SQLite: WHERE id = ? AND name = ?
|
||||
```
|
||||
|
||||
### UUID Generation
|
||||
```go
|
||||
// Postgres: RETURNING id (gen_random_uuid() DEFAULT)
|
||||
// SQLite: obj.ID = store.NewID()
|
||||
// INSERT INTO table (id, ...) VALUES (?, ...)
|
||||
// RETURNING created_at (id already set)
|
||||
```
|
||||
|
||||
### Array Types
|
||||
```go
|
||||
// Postgres: pq.Array(teamIDs) → team_id = ANY($3)
|
||||
// SQLite: for _, tid := range teamIDs { args = append(args, tid) }
|
||||
// team_id IN (?,?,?)
|
||||
//
|
||||
// Postgres: pq.Array(&tags) → stored as TEXT[]
|
||||
// SQLite: ArrayToJSON(tags) → stored as '["a","b"]'
|
||||
// ScanArray(jsonStr) → read back
|
||||
```
|
||||
|
||||
### Array Membership (granted_groups)
|
||||
```sql
|
||||
-- Postgres: gm.group_id = ANY(rg.granted_groups)
|
||||
-- SQLite: JOIN json_each(rg.granted_groups) je ON je.value = gm.group_id
|
||||
```
|
||||
|
||||
### ON CONFLICT Upserts
|
||||
```sql
|
||||
-- Postgres: ON CONFLICT (key) DO UPDATE SET value = $2, updated_by = $3
|
||||
-- SQLite: ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_by = excluded.updated_by
|
||||
```
|
||||
|
||||
### Timestamps
|
||||
```sql
|
||||
-- Postgres: NOW(), CURRENT_TIMESTAMP, TIMESTAMPTZ
|
||||
-- SQLite: datetime('now'), TEXT (ISO 8601)
|
||||
```
|
||||
|
||||
### Full-Text Search
|
||||
```sql
|
||||
-- Postgres: search_vector @@ to_tsquery('english', $1)
|
||||
-- SQLite: (title LIKE ? OR content LIKE ?) -- per word
|
||||
```
|
||||
|
||||
### Vector Search
|
||||
```
|
||||
-- Postgres: c.embedding <=> $1::vector (pgvector cosine distance)
|
||||
-- SQLite: Feature-gated. Returns error hint.
|
||||
-- KB ingestion works (stores text chunks).
|
||||
-- Future: sqlite-vec extension support.
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```bash
|
||||
# PostgreSQL (default — no changes needed)
|
||||
DB_DRIVER=postgres
|
||||
DATABASE_URL=postgres://user:pass@host:5432/switchboard
|
||||
|
||||
# SQLite
|
||||
DB_DRIVER=sqlite
|
||||
DATABASE_URL=switchboard.db # file path
|
||||
DATABASE_URL=:memory: # in-memory (testing)
|
||||
DATABASE_URL=/data/switchboard.db # absolute path
|
||||
```
|
||||
|
||||
Auto-detection: if `DB_DRIVER` is empty, inferred from `DATABASE_URL` format.
|
||||
|
||||
## SQLite Pragmas (set automatically)
|
||||
|
||||
| Pragma | Value | Purpose |
|
||||
|--------|-------|---------|
|
||||
| `journal_mode` | WAL | Concurrent readers |
|
||||
| `busy_timeout` | 5000ms | Retry on lock |
|
||||
| `foreign_keys` | ON | Enforce FK constraints |
|
||||
|
||||
## Limitations vs PostgreSQL
|
||||
|
||||
| Feature | PostgreSQL | SQLite |
|
||||
|---------|-----------|--------|
|
||||
| Vector similarity search | ✅ pgvector | ❌ Feature-gated |
|
||||
| Full-text search | ✅ tsvector/tsquery | ⚠ LIKE fallback |
|
||||
| Concurrent writes | ✅ MVCC | ⚠ Single writer (WAL) |
|
||||
| Array columns | ✅ Native | ⚠ JSON text arrays |
|
||||
| `NUMERIC(12,6)` precision | ✅ Exact | ⚠ REAL (float64) |
|
||||
| GIN indexes | ✅ Native | ❌ Not available |
|
||||
| Partial indexes | ✅ Full support | ✅ Supported |
|
||||
| RETURNING clause | ✅ Full support | ✅ SQLite 3.35+ |
|
||||
|
||||
## go.mod Addition
|
||||
|
||||
```
|
||||
require modernc.org/sqlite v1.34.5 // pure-Go, no CGO required
|
||||
```
|
||||
|
||||
## TODO (follow-up)
|
||||
|
||||
- [ ] Wire main.go dialect switch (see MAIN_GO_PATCH.md)
|
||||
- [ ] Add `modernc.org/sqlite` to go.mod
|
||||
- [ ] CI matrix: run integration tests against both drivers
|
||||
- [ ] SQLite-specific integration test suite
|
||||
- [ ] sqlite-vec extension support for vector search (optional)
|
||||
- [ ] Benchmark: SQLite vs PostgreSQL for typical workloads
|
||||
Reference in New Issue
Block a user