core/docs/archive/DESIGN-0.33.0.md

DESIGN-0.33.0 — Observability

Operate the platform without reading Go source code. Six changesets covering structured logging, Prometheus metrics, OpenAPI docs, Grafana dashboards, alerting rules, and a built-in admin dashboard.

Depends on: v0.32.0 (Multi-Replica HA, current HEAD).

Design decision: Zero new database migrations. All metrics are in-memory per-pod, scraped by Prometheus. Existing tables (provider_health, tool_health_windows, usage_log, audit_log) continue serving the admin dashboard via stores.

Library choices:

• log/slog (Go 1.22 stdlib) — zero-dep structured logging
• prometheus/client_golang v1.20+ — canonical Go Prometheus client
• Hand-curated OpenAPI 3.0.3 YAML — avoids annotation sprawl across 97 handlers
• Swagger UI v5 via CDN — single HTML embed, zero build step

---

What Already Existed

These required no structural changes, only instrumentation:

• Health accumulator — in-memory provider+tool metrics, 60s DB flush
• Hub.ConnCount() — active WebSocket connection count
• database.DB.Stats() — Go stdlib DB pool metrics
• K8s probes — /healthz/ready + /healthz/live in Helm chart
• Middleware skip paths — /metrics already in SkipPaths list
• google/uuid — already in go.mod for request ID generation

---

CS0 — Structured Logging + Request ID

Foundation changeset. All subsequent work emits structured logs.

server/logging/logger.go (new)

Init(format, level) configures the global slog logger:

• format=json → slog.NewJSONHandler(os.Stdout, opts)
• format=text → slog.NewTextHandler(os.Stdout, opts)
• Levels: debug, info, warn, error

server/middleware/request_id.go (new)

Generates UUID per request. Reuses existing google/uuid. Honors inbound X-Request-Id header for trace propagation. Sets request_id in Gin context and echoes header on response.

Middleware chain rewrite

Changed gin.Default() → gin.New() with explicit chain:

RequestID → Prometheus → Logger → Recovery → CORS

Logger middleware rewritten from log.Printf to slog.Info("request", ...) with structured fields: method, path, status, latency_ms, client_ip, request_id, user_id.

Config additions

• LogFormat (env: LOG_FORMAT, default: text)
• LogLevel (env: LOG_LEVEL, default: info)

---

CS1 — Prometheus /metrics Endpoint

server/metrics/metrics.go (new)

All metric definitions in one file using promauto:

Metric	Type	Labels
switchboard_http_requests_total	Counter	method, path_pattern, status
switchboard_http_request_duration_seconds	Histogram	method, path_pattern
switchboard_websocket_connections	Gauge	—
switchboard_completion_tokens_total	Counter	direction, model_id
switchboard_completions_total	Counter	provider_config_id, model_id, status
switchboard_completion_duration_seconds	Histogram	provider_config_id, model_id
switchboard_provider_status	Gauge	provider_config_id
switchboard_db_open_connections	Gauge	—
switchboard_db_in_use_connections	Gauge	—
switchboard_db_idle_connections	Gauge	—
switchboard_db_wait_count	Gauge	—
switchboard_db_wait_duration_seconds	Gauge	—
switchboard_task_executions_total	Counter	status
switchboard_sandbox_executions_total	Counter	entry_point, status

server/metrics/db_collector.go (new)

Background goroutine reads database.DB.Stats() every 15s, updates Prometheus gauges.

server/middleware/prometheus.go (new)

Gin middleware. Uses c.FullPath() for path_pattern label to avoid cardinality explosion from path parameters (e.g. /channels/:id not /channels/abc123).

Instrumentation points

• events/ws.go — WebSocketConnections.Inc()/Dec() on connect/disconnect
• handlers/completion.go — token counters in logUsage(), completion duration/status in recordHealth()
• handlers/stream_loop.go — same instrumentation in recordHealthFn()
• health/accumulator.go — ProviderStatus gauge updated in flush()

Route

/metrics mounted via promhttp.Handler() (no auth, standard scraping).

---

CS2 — OpenAPI Spec + Swagger UI

server/static/openapi.yaml (new)

Hand-curated OpenAPI 3.0.3 spec covering core API groups: Auth, Channels, Completions, Health, Metrics, WebSocket Tickets, Admin (stats, health, usage, dashboard).

server/static/swagger.html (new)

Minimal HTML loading Swagger UI v5 from unpkg CDN, pointing at openapi.yaml. Zero build step.

Routes

• GET /api/docs → serves embedded swagger.html
• GET /api/docs/openapi.yaml → serves embedded openapi.yaml

Both files embedded via //go:embed directives.

---

CS3 — Grafana Dashboard + Alerting Rules + Helm

Static files + Helm templates. No Go code changes.

chart/dashboards/switchboard-overview.json (new)

Grafana dashboard with 10 panels:

• Request rate, error rate, latency percentiles
• WebSocket connections, completion rate/latency by provider
• Tokens/min by model, provider status
• DB connection pool, task executions

Template variables: $datasource, $namespace, $pod.

chart/alerting/switchboard-rules.yaml (new)

Source PrometheusRule with 6 alerts:

Alert	Condition	Severity
SwitchboardPodRestart	Restarts in 1h	warning
SwitchboardProviderDown	Status > 2 for 5m	critical
SwitchboardDBPoolExhaustion	In-use/open > 80% for 5m	warning
SwitchboardHighErrorRate	5xx rate > 5% for 5m	warning
SwitchboardTaskFailureRate	Error rate > 25% for 10m	warning
SwitchboardNoCompletions	Zero completions for 15m	critical

Helm templates (new, all gated enabled: false by default)

• chart/templates/servicemonitor.yaml — ServiceMonitor CRD
• chart/templates/grafana-dashboard-configmap.yaml — ConfigMap for Grafana sidecar discovery
• chart/templates/prometheusrule.yaml — PrometheusRule CRD

Helm modifications

• chart/values.yaml — logging section + monitoring section
• chart/templates/configmap.yaml — LOG_FORMAT, LOG_LEVEL env vars
• chart/templates/services.yaml — Prometheus scrape annotations

---

CS4 — Admin Observability Dashboard

Built-in admin page for real-time health without Grafana dependency.

server/handlers/dashboard_admin.go (new)

GET /api/v1/admin/dashboard aggregating:

• Provider health summaries (from HealthStore)
• 24h usage totals (from UsageStore)
• DB pool stats (database.DB.Stats())
• WebSocket connection count (Hub.ConnCount())
• Recent errors (from AuditStore)
• Process uptime

Frontend

• SCAFFOLDING.dashboard — stat cards grid + content container
• ADMIN_LOADERS.dashboard → UI.loadAdminDashboard()
• ADMIN_SECTIONS.monitoring — dashboard added as first section
• Dashboard auto-refreshes every 30s
• CSS: provider grid, DB pool bar gauge, error list

Admin template

Monitoring tab now lands on /admin/dashboard instead of /admin/usage.

---

CS5 — Wiring, docker-compose, ICD Tests

docker-compose.yml

Added LOG_FORMAT and LOG_LEVEL env vars with defaults.

ICD tests (crud/observability.js)

Test	Assertion
GET /metrics	200, contains switchboard_* metrics
GET /api/docs	200, contains swagger-ui
GET /api/docs/openapi.yaml	200, contains openapi: and paths:
X-Request-Id generation	Response includes 36-char UUID header
X-Request-Id passthrough	Client-sent header echoed back
GET /admin/dashboard	200, has uptime, ws_connections, provider_health

---

Verification

1. LOG_FORMAT=json docker compose up → JSON log lines on stdout
2. curl localhost:8080/metrics | grep switchboard_ → all metrics present
3. curl -I localhost:8080/api/v1/channels → X-Request-Id header
4. /api/docs in browser → Swagger UI renders
5. /admin → Monitoring → Dashboard shows live stat cards
6. helm lint chart/ --set monitoring.serviceMonitor.enabled=true → passes
7. ICD test suite passes (existing + observability tests)