Changeset 0.33.0 (#207)
Co-authored-by: Jeffrey Smith <jasafpro@gmail.com> Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
This commit is contained in:
238
docs/DESIGN-0.33.0.md
Normal file
238
docs/DESIGN-0.33.0.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# DESIGN-0.33.0 — Observability
|
||||
|
||||
Operate the platform without reading Go source code. Six changesets
|
||||
covering structured logging, Prometheus metrics, OpenAPI docs, Grafana
|
||||
dashboards, alerting rules, and a built-in admin dashboard.
|
||||
|
||||
Depends on: v0.32.0 (Multi-Replica HA, current HEAD).
|
||||
|
||||
**Design decision:** Zero new database migrations. All metrics are
|
||||
in-memory per-pod, scraped by Prometheus. Existing tables
|
||||
(`provider_health`, `tool_health_windows`, `usage_log`, `audit_log`)
|
||||
continue serving the admin dashboard via stores.
|
||||
|
||||
**Library choices:**
|
||||
- `log/slog` (Go 1.22 stdlib) — zero-dep structured logging
|
||||
- `prometheus/client_golang` v1.20+ — canonical Go Prometheus client
|
||||
- Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
|
||||
- Swagger UI v5 via CDN — single HTML embed, zero build step
|
||||
|
||||
---
|
||||
|
||||
## What Already Existed
|
||||
|
||||
These required no structural changes, only instrumentation:
|
||||
|
||||
- **Health accumulator** — in-memory provider+tool metrics, 60s DB flush
|
||||
- **`Hub.ConnCount()`** — active WebSocket connection count
|
||||
- **`database.DB.Stats()`** — Go stdlib DB pool metrics
|
||||
- **K8s probes** — `/healthz/ready` + `/healthz/live` in Helm chart
|
||||
- **Middleware skip paths** — `/metrics` already in `SkipPaths` list
|
||||
- **`google/uuid`** — already in go.mod for request ID generation
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Structured Logging + Request ID
|
||||
|
||||
Foundation changeset. All subsequent work emits structured logs.
|
||||
|
||||
### `server/logging/logger.go` (new)
|
||||
|
||||
`Init(format, level)` configures the global `slog` logger:
|
||||
- `format=json` → `slog.NewJSONHandler(os.Stdout, opts)`
|
||||
- `format=text` → `slog.NewTextHandler(os.Stdout, opts)`
|
||||
- Levels: debug, info, warn, error
|
||||
|
||||
### `server/middleware/request_id.go` (new)
|
||||
|
||||
Generates UUID per request. Reuses existing `google/uuid`.
|
||||
Honors inbound `X-Request-Id` header for trace propagation.
|
||||
Sets `request_id` in Gin context and echoes header on response.
|
||||
|
||||
### Middleware chain rewrite
|
||||
|
||||
Changed `gin.Default()` → `gin.New()` with explicit chain:
|
||||
|
||||
```
|
||||
RequestID → Prometheus → Logger → Recovery → CORS
|
||||
```
|
||||
|
||||
Logger middleware rewritten from `log.Printf` to `slog.Info("request", ...)`
|
||||
with structured fields: method, path, status, latency_ms, client_ip,
|
||||
request_id, user_id.
|
||||
|
||||
### Config additions
|
||||
|
||||
- `LogFormat` (env: `LOG_FORMAT`, default: `text`)
|
||||
- `LogLevel` (env: `LOG_LEVEL`, default: `info`)
|
||||
|
||||
---
|
||||
|
||||
## CS1 — Prometheus `/metrics` Endpoint
|
||||
|
||||
### `server/metrics/metrics.go` (new)
|
||||
|
||||
All metric definitions in one file using `promauto`:
|
||||
|
||||
| Metric | Type | Labels |
|
||||
|--------|------|--------|
|
||||
| `switchboard_http_requests_total` | Counter | method, path_pattern, status |
|
||||
| `switchboard_http_request_duration_seconds` | Histogram | method, path_pattern |
|
||||
| `switchboard_websocket_connections` | Gauge | — |
|
||||
| `switchboard_completion_tokens_total` | Counter | direction, model_id |
|
||||
| `switchboard_completions_total` | Counter | provider_config_id, model_id, status |
|
||||
| `switchboard_completion_duration_seconds` | Histogram | provider_config_id, model_id |
|
||||
| `switchboard_provider_status` | Gauge | provider_config_id |
|
||||
| `switchboard_db_open_connections` | Gauge | — |
|
||||
| `switchboard_db_in_use_connections` | Gauge | — |
|
||||
| `switchboard_db_idle_connections` | Gauge | — |
|
||||
| `switchboard_db_wait_count` | Gauge | — |
|
||||
| `switchboard_db_wait_duration_seconds` | Gauge | — |
|
||||
| `switchboard_task_executions_total` | Counter | status |
|
||||
| `switchboard_sandbox_executions_total` | Counter | entry_point, status |
|
||||
|
||||
### `server/metrics/db_collector.go` (new)
|
||||
|
||||
Background goroutine reads `database.DB.Stats()` every 15s, updates
|
||||
Prometheus gauges.
|
||||
|
||||
### `server/middleware/prometheus.go` (new)
|
||||
|
||||
Gin middleware. Uses `c.FullPath()` for `path_pattern` label to avoid
|
||||
cardinality explosion from path parameters (e.g. `/channels/:id`
|
||||
not `/channels/abc123`).
|
||||
|
||||
### Instrumentation points
|
||||
|
||||
- `events/ws.go` — `WebSocketConnections.Inc()/Dec()` on connect/disconnect
|
||||
- `handlers/completion.go` — token counters in `logUsage()`, completion
|
||||
duration/status in `recordHealth()`
|
||||
- `handlers/stream_loop.go` — same instrumentation in `recordHealthFn()`
|
||||
- `health/accumulator.go` — `ProviderStatus` gauge updated in `flush()`
|
||||
|
||||
### Route
|
||||
|
||||
`/metrics` mounted via `promhttp.Handler()` (no auth, standard scraping).
|
||||
|
||||
---
|
||||
|
||||
## CS2 — OpenAPI Spec + Swagger UI
|
||||
|
||||
### `server/static/openapi.yaml` (new)
|
||||
|
||||
Hand-curated OpenAPI 3.0.3 spec covering core API groups:
|
||||
Auth, Channels, Completions, Health, Metrics, WebSocket Tickets,
|
||||
Admin (stats, health, usage, dashboard).
|
||||
|
||||
### `server/static/swagger.html` (new)
|
||||
|
||||
Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at
|
||||
`openapi.yaml`. Zero build step.
|
||||
|
||||
### Routes
|
||||
|
||||
- `GET /api/docs` → serves embedded `swagger.html`
|
||||
- `GET /api/docs/openapi.yaml` → serves embedded `openapi.yaml`
|
||||
|
||||
Both files embedded via `//go:embed` directives.
|
||||
|
||||
---
|
||||
|
||||
## CS3 — Grafana Dashboard + Alerting Rules + Helm
|
||||
|
||||
Static files + Helm templates. No Go code changes.
|
||||
|
||||
### `chart/dashboards/switchboard-overview.json` (new)
|
||||
|
||||
Grafana dashboard with 10 panels:
|
||||
- Request rate, error rate, latency percentiles
|
||||
- WebSocket connections, completion rate/latency by provider
|
||||
- Tokens/min by model, provider status
|
||||
- DB connection pool, task executions
|
||||
|
||||
Template variables: `$datasource`, `$namespace`, `$pod`.
|
||||
|
||||
### `chart/alerting/switchboard-rules.yaml` (new)
|
||||
|
||||
Source PrometheusRule with 6 alerts:
|
||||
|
||||
| Alert | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| SwitchboardPodRestart | Restarts in 1h | warning |
|
||||
| SwitchboardProviderDown | Status > 2 for 5m | critical |
|
||||
| SwitchboardDBPoolExhaustion | In-use/open > 80% for 5m | warning |
|
||||
| SwitchboardHighErrorRate | 5xx rate > 5% for 5m | warning |
|
||||
| SwitchboardTaskFailureRate | Error rate > 25% for 10m | warning |
|
||||
| SwitchboardNoCompletions | Zero completions for 15m | critical |
|
||||
|
||||
### Helm templates (new, all gated `enabled: false` by default)
|
||||
|
||||
- `chart/templates/servicemonitor.yaml` — ServiceMonitor CRD
|
||||
- `chart/templates/grafana-dashboard-configmap.yaml` — ConfigMap for
|
||||
Grafana sidecar discovery
|
||||
- `chart/templates/prometheusrule.yaml` — PrometheusRule CRD
|
||||
|
||||
### Helm modifications
|
||||
|
||||
- `chart/values.yaml` — `logging` section + `monitoring` section
|
||||
- `chart/templates/configmap.yaml` — `LOG_FORMAT`, `LOG_LEVEL` env vars
|
||||
- `chart/templates/services.yaml` — Prometheus scrape annotations
|
||||
|
||||
---
|
||||
|
||||
## CS4 — Admin Observability Dashboard
|
||||
|
||||
Built-in admin page for real-time health without Grafana dependency.
|
||||
|
||||
### `server/handlers/dashboard_admin.go` (new)
|
||||
|
||||
`GET /api/v1/admin/dashboard` aggregating:
|
||||
- Provider health summaries (from `HealthStore`)
|
||||
- 24h usage totals (from `UsageStore`)
|
||||
- DB pool stats (`database.DB.Stats()`)
|
||||
- WebSocket connection count (`Hub.ConnCount()`)
|
||||
- Recent errors (from `AuditStore`)
|
||||
- Process uptime
|
||||
|
||||
### Frontend
|
||||
|
||||
- `SCAFFOLDING.dashboard` — stat cards grid + content container
|
||||
- `ADMIN_LOADERS.dashboard` → `UI.loadAdminDashboard()`
|
||||
- `ADMIN_SECTIONS.monitoring` — `dashboard` added as first section
|
||||
- Dashboard auto-refreshes every 30s
|
||||
- CSS: provider grid, DB pool bar gauge, error list
|
||||
|
||||
### Admin template
|
||||
|
||||
Monitoring tab now lands on `/admin/dashboard` instead of `/admin/usage`.
|
||||
|
||||
---
|
||||
|
||||
## CS5 — Wiring, docker-compose, ICD Tests
|
||||
|
||||
### docker-compose.yml
|
||||
|
||||
Added `LOG_FORMAT` and `LOG_LEVEL` env vars with defaults.
|
||||
|
||||
### ICD tests (`crud/observability.js`)
|
||||
|
||||
| Test | Assertion |
|
||||
|------|-----------|
|
||||
| `GET /metrics` | 200, contains `switchboard_*` metrics |
|
||||
| `GET /api/docs` | 200, contains `swagger-ui` |
|
||||
| `GET /api/docs/openapi.yaml` | 200, contains `openapi:` and `paths:` |
|
||||
| X-Request-Id generation | Response includes 36-char UUID header |
|
||||
| X-Request-Id passthrough | Client-sent header echoed back |
|
||||
| `GET /admin/dashboard` | 200, has `uptime`, `ws_connections`, `provider_health` |
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. `LOG_FORMAT=json docker compose up` → JSON log lines on stdout
|
||||
2. `curl localhost:8080/metrics | grep switchboard_` → all metrics present
|
||||
3. `curl -I localhost:8080/api/v1/channels` → `X-Request-Id` header
|
||||
4. `/api/docs` in browser → Swagger UI renders
|
||||
5. `/admin` → Monitoring → Dashboard shows live stat cards
|
||||
6. `helm lint chart/ --set monitoring.serviceMonitor.enabled=true` → passes
|
||||
7. ICD test suite passes (existing + observability tests)
|
||||
@@ -31,7 +31,7 @@ v0.9.x–v0.28.7 Foundation through Platform Polish ✅
|
||||
Extension Track Operations Track
|
||||
│ │
|
||||
v0.29.0 Starlark Sandbox ✅ v0.32.0 Multi-Replica HA ✅
|
||||
v0.29.1 API Extensions ✅ v0.33.0 Observability
|
||||
v0.29.1 API Extensions ✅ v0.33.0 Observability ✅
|
||||
v0.29.2 DB Extensions ✅ v0.34.0 Data Portability
|
||||
v0.29.3 Workflow Forms ✅ │
|
||||
v0.30.0 Package Lifecycle ✅ │
|
||||
@@ -43,6 +43,8 @@ v0.9.x–v0.28.7 Foundation through Platform Polish ✅
|
||||
│ │
|
||||
│ v0.35.0 Workflow Product
|
||||
│ │
|
||||
│ v0.36.0 Full OpenAPI Spec
|
||||
│ │
|
||||
══════╪═══════════════════════════════╪══════
|
||||
│ MVP v0.50.0 │
|
||||
│ (v0.31.2 + v0.35.0 │
|
||||
@@ -384,23 +386,33 @@ no new coordination infrastructure. See `docs/DESIGN-0.32.0.md`.
|
||||
|
||||
**Schema:** `020_ha.sql` — `ws_tickets`, `rate_limit_counters`.
|
||||
|
||||
### v0.33.0 — Observability
|
||||
### v0.33.0 — Observability ✅
|
||||
|
||||
Metrics, dashboards, alerting. Operate the platform without reading
|
||||
Go source code.
|
||||
|
||||
Depends on: v0.32.0 (multi-replica metrics aggregation).
|
||||
|
||||
- [ ] Prometheus `/metrics` endpoint: request latency, active WebSocket
|
||||
gauge, completion token counters, DB pool stats, provider health
|
||||
- [ ] Grafana dashboard template: system overview, per-team usage,
|
||||
provider latency, error rates
|
||||
- [ ] Structured logging: `LOG_FORMAT=json`, request ID propagation,
|
||||
correlation IDs in completion chains
|
||||
- [ ] Alerting rules: OOM recovery, provider down, pool exhaustion,
|
||||
task failure rate
|
||||
- [ ] Admin dashboard surface: real-time health (built-in, no Grafana)
|
||||
- [ ] Swagger/OpenAPI: auto-generated spec from route definitions, served at /api/docs
|
||||
**Delivered (6 changesets):**
|
||||
- [x] Structured logging (`slog`): `LOG_FORMAT=json|text`,
|
||||
`LOG_LEVEL=debug|info|warn|error`. Request ID middleware
|
||||
(`X-Request-Id`), correlation IDs through completion chain.
|
||||
- [x] Prometheus `/metrics` endpoint: HTTP request counters/histograms,
|
||||
WebSocket gauge, completion tokens/duration, provider status,
|
||||
DB pool stats, task/sandbox execution counters.
|
||||
- [x] OpenAPI 3.0.3 spec (hand-curated, core API groups) + Swagger UI
|
||||
at `/api/docs` with system dark/light mode, WCAG AA contrast.
|
||||
- [x] Grafana dashboard JSON: request rate, latency percentiles,
|
||||
provider health, token usage, DB pool, Go runtime.
|
||||
- [x] PrometheusRule alerting: OOM recovery, provider down, pool
|
||||
exhaustion, high error rate, task failure spike.
|
||||
- [x] Admin monitoring dashboard: provider health, 24h usage, DB pool,
|
||||
WS connections, Go runtime stats, storage status, recent errors,
|
||||
30s auto-refresh.
|
||||
|
||||
**Deferred to v0.36.0:**
|
||||
- Full OpenAPI spec coverage (all 20 ICD domains)
|
||||
- Bearer token / mTLS auth documentation in spec
|
||||
|
||||
### v0.34.0 — Data Portability
|
||||
|
||||
@@ -472,6 +484,48 @@ Depends on: v0.31.2 (team workflow self-service), v0.29.0 (Starlark sandbox).
|
||||
- Data intake: form → Starlark enrichment → human review → follow-up
|
||||
- Onboarding: multi-stage form → conditional branching → team assignment
|
||||
|
||||
### v0.36.0 — Full OpenAPI Spec
|
||||
|
||||
Complete OpenAPI 3.0.3 coverage for every API domain. Expand the
|
||||
hand-curated spec from v0.33.0 (core groups only) to cover all 20 ICD
|
||||
test domains. Document Bearer token auth and mTLS modes.
|
||||
|
||||
Depends on: v0.33.0 (Swagger UI + initial spec).
|
||||
|
||||
**API domains to document (matching ICD test suite):**
|
||||
- [ ] Admin (system settings, stats, storage, users, provider config)
|
||||
- [ ] Channels (full CRUD + membership, already partially covered)
|
||||
- [ ] Completions (streaming + non-streaming, already partially covered)
|
||||
- [ ] Extensions (package install, permissions, lifecycle)
|
||||
- [ ] Knowledge (KB articles, sources, search, embeddings)
|
||||
- [ ] Memory (conversation memory, compaction, search)
|
||||
- [ ] Models (provider models, BYOK, capability matching)
|
||||
- [ ] Notes (CRUD, graph links, search)
|
||||
- [ ] Notifications (list, mark read, preferences)
|
||||
- [ ] Personas (CRUD, system prompts, tool config)
|
||||
- [ ] Profile (user profile, preferences, avatar)
|
||||
- [ ] Projects (CRUD, membership, file uploads)
|
||||
- [ ] Surfaces (extension surfaces, mounting, lifecycle)
|
||||
- [ ] Tasks (scheduled tasks, execution history, Starlark tasks)
|
||||
- [ ] Teams (CRUD, membership, roles, slugs)
|
||||
- [ ] Team Workflows (team-scoped CRUD, stages, publish)
|
||||
- [ ] Workflows (platform-level CRUD, stages, instances, assignments)
|
||||
- [ ] Workspaces (CRUD, membership, settings)
|
||||
- [ ] Dashboard Package (package-specific API routes)
|
||||
- [ ] Editor Package (package-specific API routes)
|
||||
|
||||
**Auth documentation:**
|
||||
- [ ] Bearer JWT flow (login → token → `Authorization: Bearer <token>`)
|
||||
- [ ] mTLS mode (NPE-to-NPE, no Bearer needed)
|
||||
- [ ] API key mode (for service-to-service integrations)
|
||||
- [ ] Security scheme definitions in OpenAPI spec
|
||||
|
||||
**Quality:**
|
||||
- [ ] Request/response schemas with examples for every endpoint
|
||||
- [ ] Error response schemas (400, 401, 403, 404, 409, 422, 429, 500)
|
||||
- [ ] Pagination parameters documented consistently
|
||||
- [ ] WebSocket event schemas (connection, ticket, event types)
|
||||
|
||||
---
|
||||
|
||||
## MVP v0.50.0
|
||||
@@ -487,6 +541,7 @@ reading Go source code. Team admins build workflows visually.
|
||||
data portability)
|
||||
- Workflow product v0.35.0 (form rendering, data pipeline, conditional
|
||||
routing, structured review, monitoring dashboard)
|
||||
- Full OpenAPI spec v0.36.0 (complete API documentation for all domains)
|
||||
|
||||
**Additionally requires:**
|
||||
- [ ] Deployment guide: step-by-step for IT team (PG cluster,
|
||||
|
||||
Reference in New Issue
Block a user