239 lines
8.1 KiB
Markdown
239 lines
8.1 KiB
Markdown
# DESIGN-0.33.0 — Observability
|
|
|
|
Operate the platform without reading Go source code. Six changesets
|
|
covering structured logging, Prometheus metrics, OpenAPI docs, Grafana
|
|
dashboards, alerting rules, and a built-in admin dashboard.
|
|
|
|
Depends on: v0.32.0 (Multi-Replica HA, current HEAD).
|
|
|
|
**Design decision:** Zero new database migrations. All metrics are
|
|
in-memory per-pod, scraped by Prometheus. Existing tables
|
|
(`provider_health`, `tool_health_windows`, `usage_log`, `audit_log`)
|
|
continue serving the admin dashboard via stores.
|
|
|
|
**Library choices:**
|
|
- `log/slog` (Go 1.22 stdlib) — zero-dep structured logging
|
|
- `prometheus/client_golang` v1.20+ — canonical Go Prometheus client
|
|
- Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
|
|
- Swagger UI v5 via CDN — single HTML embed, zero build step
|
|
|
|
---
|
|
|
|
## What Already Existed
|
|
|
|
These required no structural changes, only instrumentation:
|
|
|
|
- **Health accumulator** — in-memory provider+tool metrics, 60s DB flush
|
|
- **`Hub.ConnCount()`** — active WebSocket connection count
|
|
- **`database.DB.Stats()`** — Go stdlib DB pool metrics
|
|
- **K8s probes** — `/healthz/ready` + `/healthz/live` in Helm chart
|
|
- **Middleware skip paths** — `/metrics` already in `SkipPaths` list
|
|
- **`google/uuid`** — already in go.mod for request ID generation
|
|
|
|
---
|
|
|
|
## CS0 — Structured Logging + Request ID
|
|
|
|
Foundation changeset. All subsequent work emits structured logs.
|
|
|
|
### `server/logging/logger.go` (new)
|
|
|
|
`Init(format, level)` configures the global `slog` logger:
|
|
- `format=json` → `slog.NewJSONHandler(os.Stdout, opts)`
|
|
- `format=text` → `slog.NewTextHandler(os.Stdout, opts)`
|
|
- Levels: debug, info, warn, error
|
|
|
|
### `server/middleware/request_id.go` (new)
|
|
|
|
Generates UUID per request. Reuses existing `google/uuid`.
|
|
Honors inbound `X-Request-Id` header for trace propagation.
|
|
Sets `request_id` in Gin context and echoes header on response.
|
|
|
|
### Middleware chain rewrite
|
|
|
|
Changed `gin.Default()` → `gin.New()` with explicit chain:
|
|
|
|
```
|
|
RequestID → Prometheus → Logger → Recovery → CORS
|
|
```
|
|
|
|
Logger middleware rewritten from `log.Printf` to `slog.Info("request", ...)`
|
|
with structured fields: method, path, status, latency_ms, client_ip,
|
|
request_id, user_id.
|
|
|
|
### Config additions
|
|
|
|
- `LogFormat` (env: `LOG_FORMAT`, default: `text`)
|
|
- `LogLevel` (env: `LOG_LEVEL`, default: `info`)
|
|
|
|
---
|
|
|
|
## CS1 — Prometheus `/metrics` Endpoint
|
|
|
|
### `server/metrics/metrics.go` (new)
|
|
|
|
All metric definitions in one file using `promauto`:
|
|
|
|
| Metric | Type | Labels |
|
|
|--------|------|--------|
|
|
| `switchboard_http_requests_total` | Counter | method, path_pattern, status |
|
|
| `switchboard_http_request_duration_seconds` | Histogram | method, path_pattern |
|
|
| `switchboard_websocket_connections` | Gauge | — |
|
|
| `switchboard_completion_tokens_total` | Counter | direction, model_id |
|
|
| `switchboard_completions_total` | Counter | provider_config_id, model_id, status |
|
|
| `switchboard_completion_duration_seconds` | Histogram | provider_config_id, model_id |
|
|
| `switchboard_provider_status` | Gauge | provider_config_id |
|
|
| `switchboard_db_open_connections` | Gauge | — |
|
|
| `switchboard_db_in_use_connections` | Gauge | — |
|
|
| `switchboard_db_idle_connections` | Gauge | — |
|
|
| `switchboard_db_wait_count` | Gauge | — |
|
|
| `switchboard_db_wait_duration_seconds` | Gauge | — |
|
|
| `switchboard_task_executions_total` | Counter | status |
|
|
| `switchboard_sandbox_executions_total` | Counter | entry_point, status |
|
|
|
|
### `server/metrics/db_collector.go` (new)
|
|
|
|
Background goroutine reads `database.DB.Stats()` every 15s, updates
|
|
Prometheus gauges.
|
|
|
|
### `server/middleware/prometheus.go` (new)
|
|
|
|
Gin middleware. Uses `c.FullPath()` for `path_pattern` label to avoid
|
|
cardinality explosion from path parameters (e.g. `/channels/:id`
|
|
not `/channels/abc123`).
|
|
|
|
### Instrumentation points
|
|
|
|
- `events/ws.go` — `WebSocketConnections.Inc()/Dec()` on connect/disconnect
|
|
- `handlers/completion.go` — token counters in `logUsage()`, completion
|
|
duration/status in `recordHealth()`
|
|
- `handlers/stream_loop.go` — same instrumentation in `recordHealthFn()`
|
|
- `health/accumulator.go` — `ProviderStatus` gauge updated in `flush()`
|
|
|
|
### Route
|
|
|
|
`/metrics` mounted via `promhttp.Handler()` (no auth, standard scraping).
|
|
|
|
---
|
|
|
|
## CS2 — OpenAPI Spec + Swagger UI
|
|
|
|
### `server/static/openapi.yaml` (new)
|
|
|
|
Hand-curated OpenAPI 3.0.3 spec covering core API groups:
|
|
Auth, Channels, Completions, Health, Metrics, WebSocket Tickets,
|
|
Admin (stats, health, usage, dashboard).
|
|
|
|
### `server/static/swagger.html` (new)
|
|
|
|
Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at
|
|
`openapi.yaml`. Zero build step.
|
|
|
|
### Routes
|
|
|
|
- `GET /api/docs` → serves embedded `swagger.html`
|
|
- `GET /api/docs/openapi.yaml` → serves embedded `openapi.yaml`
|
|
|
|
Both files embedded via `//go:embed` directives.
|
|
|
|
---
|
|
|
|
## CS3 — Grafana Dashboard + Alerting Rules + Helm
|
|
|
|
Static files + Helm templates. No Go code changes.
|
|
|
|
### `chart/dashboards/switchboard-overview.json` (new)
|
|
|
|
Grafana dashboard with 10 panels:
|
|
- Request rate, error rate, latency percentiles
|
|
- WebSocket connections, completion rate/latency by provider
|
|
- Tokens/min by model, provider status
|
|
- DB connection pool, task executions
|
|
|
|
Template variables: `$datasource`, `$namespace`, `$pod`.
|
|
|
|
### `chart/alerting/switchboard-rules.yaml` (new)
|
|
|
|
Source PrometheusRule with 6 alerts:
|
|
|
|
| Alert | Condition | Severity |
|
|
|-------|-----------|----------|
|
|
| SwitchboardPodRestart | Restarts in 1h | warning |
|
|
| SwitchboardProviderDown | Status > 2 for 5m | critical |
|
|
| SwitchboardDBPoolExhaustion | In-use/open > 80% for 5m | warning |
|
|
| SwitchboardHighErrorRate | 5xx rate > 5% for 5m | warning |
|
|
| SwitchboardTaskFailureRate | Error rate > 25% for 10m | warning |
|
|
| SwitchboardNoCompletions | Zero completions for 15m | critical |
|
|
|
|
### Helm templates (new, all gated `enabled: false` by default)
|
|
|
|
- `chart/templates/servicemonitor.yaml` — ServiceMonitor CRD
|
|
- `chart/templates/grafana-dashboard-configmap.yaml` — ConfigMap for
|
|
Grafana sidecar discovery
|
|
- `chart/templates/prometheusrule.yaml` — PrometheusRule CRD
|
|
|
|
### Helm modifications
|
|
|
|
- `chart/values.yaml` — `logging` section + `monitoring` section
|
|
- `chart/templates/configmap.yaml` — `LOG_FORMAT`, `LOG_LEVEL` env vars
|
|
- `chart/templates/services.yaml` — Prometheus scrape annotations
|
|
|
|
---
|
|
|
|
## CS4 — Admin Observability Dashboard
|
|
|
|
Built-in admin page for real-time health without Grafana dependency.
|
|
|
|
### `server/handlers/dashboard_admin.go` (new)
|
|
|
|
`GET /api/v1/admin/dashboard` aggregating:
|
|
- Provider health summaries (from `HealthStore`)
|
|
- 24h usage totals (from `UsageStore`)
|
|
- DB pool stats (`database.DB.Stats()`)
|
|
- WebSocket connection count (`Hub.ConnCount()`)
|
|
- Recent errors (from `AuditStore`)
|
|
- Process uptime
|
|
|
|
### Frontend
|
|
|
|
- `SCAFFOLDING.dashboard` — stat cards grid + content container
|
|
- `ADMIN_LOADERS.dashboard` → `UI.loadAdminDashboard()`
|
|
- `ADMIN_SECTIONS.monitoring` — `dashboard` added as first section
|
|
- Dashboard auto-refreshes every 30s
|
|
- CSS: provider grid, DB pool bar gauge, error list
|
|
|
|
### Admin template
|
|
|
|
Monitoring tab now lands on `/admin/dashboard` instead of `/admin/usage`.
|
|
|
|
---
|
|
|
|
## CS5 — Wiring, docker-compose, ICD Tests
|
|
|
|
### docker-compose.yml
|
|
|
|
Added `LOG_FORMAT` and `LOG_LEVEL` env vars with defaults.
|
|
|
|
### ICD tests (`crud/observability.js`)
|
|
|
|
| Test | Assertion |
|
|
|------|-----------|
|
|
| `GET /metrics` | 200, contains `switchboard_*` metrics |
|
|
| `GET /api/docs` | 200, contains `swagger-ui` |
|
|
| `GET /api/docs/openapi.yaml` | 200, contains `openapi:` and `paths:` |
|
|
| X-Request-Id generation | Response includes 36-char UUID header |
|
|
| X-Request-Id passthrough | Client-sent header echoed back |
|
|
| `GET /admin/dashboard` | 200, has `uptime`, `ws_connections`, `provider_health` |
|
|
|
|
---
|
|
|
|
## Verification
|
|
|
|
1. `LOG_FORMAT=json docker compose up` → JSON log lines on stdout
|
|
2. `curl localhost:8080/metrics | grep switchboard_` → all metrics present
|
|
3. `curl -I localhost:8080/api/v1/channels` → `X-Request-Id` header
|
|
4. `/api/docs` in browser → Swagger UI renders
|
|
5. `/admin` → Monitoring → Dashboard shows live stat cards
|
|
6. `helm lint chart/ --set monitoring.serviceMonitor.enabled=true` → passes
|
|
7. ICD test suite passes (existing + observability tests)
|