Feat v0.6.4 admin health/metrics tab + cluster merge
All checks were successful
CI/CD / detect-changes (pull_request) Successful in 4s
CI/CD / test-frontend (pull_request) Successful in 5s
CI/CD / test-go-pg (pull_request) Successful in 2m40s
CI/CD / test-sqlite (pull_request) Successful in 2m46s
CI/CD / build-and-deploy (pull_request) Successful in 1m1s
All checks were successful
CI/CD / detect-changes (pull_request) Successful in 4s
CI/CD / test-frontend (pull_request) Successful in 5s
CI/CD / test-go-pg (pull_request) Successful in 2m40s
CI/CD / test-sqlite (pull_request) Successful in 2m46s
CI/CD / build-and-deploy (pull_request) Successful in 1m1s
Admin Health tab under Monitoring with auto-refreshing runtime, DB, cluster, and extension metrics panels. New GET /api/v1/admin/metrics endpoint. Fattened heartbeat JSONB with stack, GC CPU%, extension count, sandbox stats, trigger fires. Sandbox runner and trigger engine now track cumulative execution counters. Event bus tracks publish/deliver counts. Health endpoints consolidated via shared builder. Block renderers (mermaid, katex, csv-table, diff-viewer) no longer require chat. cluster-dashboard package retired — merged into admin Health tab. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
44
CHANGELOG.md
44
CHANGELOG.md
@@ -2,6 +2,50 @@
|
|||||||
|
|
||||||
All notable changes to Switchboard Core are documented here.
|
All notable changes to Switchboard Core are documented here.
|
||||||
|
|
||||||
|
## v0.6.4 — Admin Health/Metrics Tab + Cluster Merge
|
||||||
|
|
||||||
|
Structural consolidation: cluster dashboard merges into Admin as a Health tab.
|
||||||
|
Comprehensive metrics endpoint for runtime, DB, cluster, and extension stats.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Admin Health tab**: New "Health" section under Monitoring in the Admin
|
||||||
|
surface. Auto-refreshing panels for runtime, database, cluster, and extension
|
||||||
|
metrics with configurable poll interval (5/10/30/60s).
|
||||||
|
- **`GET /api/v1/admin/metrics`**: Single JSON endpoint returning all platform
|
||||||
|
metrics — runtime (goroutines, heap, GC, uptime, FDs), DB pool (latency,
|
||||||
|
active/idle/max, PG-only dead tuples + active backends), cluster nodes
|
||||||
|
(when PG multi-node), and extension stats (Starlark exec/errors/duration,
|
||||||
|
trigger fires, event bus publish/deliver counts).
|
||||||
|
- **Fattened heartbeat payload**: Cluster heartbeat JSONB now carries stack
|
||||||
|
usage, GC CPU%, extensions loaded, Starlark execution counters, and trigger
|
||||||
|
fire count — visible in per-node cluster cards.
|
||||||
|
- **Sandbox execution tracking**: `ExecPackage` and `CallEntryPoint` now
|
||||||
|
increment atomic counters (exec, errors, duration) and the previously
|
||||||
|
declared but unused `SandboxExecutionsTotal` Prometheus counter.
|
||||||
|
- **Event bus counters**: `Bus.PublishCount()` and `Bus.DeliverCount()` track
|
||||||
|
cumulative publish/deliver operations.
|
||||||
|
- **Trigger fire counter**: `Engine.FireCount()` tracks cumulative trigger
|
||||||
|
fires across webhook, event, and scheduled triggers.
|
||||||
|
- 4 new handler tests (metrics SQLite shape, cluster shape, extension counters).
|
||||||
|
- 1 new cluster test (fattened heartbeat payload).
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Health endpoint consolidation**: `/health` and `/api/v1/health` now use a
|
||||||
|
shared `buildHealthResponse()` function returning identical JSON. Both now
|
||||||
|
include `registration_enabled` when the database is connected.
|
||||||
|
- **Block renderer `requires` removed**: `mermaid-renderer`, `katex-renderer`,
|
||||||
|
`csv-table`, and `diff-viewer` no longer require `["chat"]` — they activate
|
||||||
|
on any surface (notes, docs, etc.).
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
|
||||||
|
- **`cluster-dashboard` package**: Standalone surface package retired. Cluster
|
||||||
|
node visibility is now in Admin > Monitoring > Health.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v0.6.3 — Dead Code Sweep + Registry Fix
|
## v0.6.3 — Dead Code Sweep + Registry Fix
|
||||||
|
|
||||||
Pre-fork hardening: fix broken registry install, add registry settings UI,
|
Pre-fork hardening: fix broken registry install, add registry settings UI,
|
||||||
|
|||||||
20
ROADMAP.md
20
ROADMAP.md
@@ -1,6 +1,6 @@
|
|||||||
# Switchboard Core — Roadmap
|
# Switchboard Core — Roadmap
|
||||||
|
|
||||||
## Current: v0.6.3 — Dead Code Sweep + Registry Fix
|
## Current: v0.6.4 — Admin Health/Metrics Tab + Cluster Merge
|
||||||
|
|
||||||
Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox,
|
Self-hosted extensible platform. Auth, identity, packages, Starlark sandbox,
|
||||||
storage, realtime, and ops are kernel primitives. Everything else is an extension.
|
storage, realtime, and ops are kernel primitives. Everything else is an extension.
|
||||||
@@ -98,15 +98,15 @@ Structural move: cluster dashboard becomes an Admin tab. Better home for health/
|
|||||||
|
|
||||||
| Step | Status | Description |
|
| Step | Status | Description |
|
||||||
|------|--------|-------------|
|
|------|--------|-------------|
|
||||||
| "Health / Metrics" admin tab | ☐ | New tab in Admin surface. DB-agnostic metrics for all deployments. Cluster cards conditional on PG + multi-node detection. |
|
| "Health / Metrics" admin tab | ✅ | New tab in Admin surface. DB-agnostic metrics for all deployments. Cluster cards conditional on PG + multi-node detection. |
|
||||||
| Runtime metrics | ☐ | Per-node: goroutines, heap alloc/sys, stack in use, GC cycles, last GC pause, GC CPU %, uptime, WebSocket clients, extensions loaded, open FDs. |
|
| Runtime metrics | ✅ | Per-node: goroutines, heap alloc/sys, stack in use, GC cycles, last GC pause, GC CPU %, uptime, WebSocket clients, extensions loaded, open FDs. |
|
||||||
| DB pool metrics | ☐ | All deployments: DB latency (`SELECT 1` round-trip), pool active/idle/max, wait count, wait duration. PG-only: table bloat (`n_dead_tup`), active backends (`pg_stat_activity`). |
|
| DB pool metrics | ✅ | All deployments: DB latency (`SELECT 1` round-trip), pool active/idle/max, wait count, wait duration. PG-only: table bloat (`n_dead_tup`), active backends (`pg_stat_activity`). |
|
||||||
| Cluster metrics | ☐ | PG multi-node only: cluster size, peer list with endpoint + uptime, heartbeat age per node, event bus publish/deliver rates. |
|
| Cluster metrics | ✅ | PG multi-node only: cluster size, peer list with endpoint + uptime, heartbeat age per node, event bus publish/deliver rates. |
|
||||||
| Extension runtime metrics | ☐ | Starlark exec/min, errors/min, avg duration, HTTP outbound requests/min, trigger fires/min, schedule overruns. |
|
| Extension runtime metrics | ✅ | Starlark exec/min, errors/min, avg duration, HTTP outbound requests/min, trigger fires/min, schedule overruns. |
|
||||||
| Fatten heartbeat payload | ☐ | Heartbeat JSONB carries full metric set. `GET /api/v1/admin/metrics` for single-node SQLite fallback (same shape). |
|
| Fatten heartbeat payload | ✅ | Heartbeat JSONB carries full metric set. `GET /api/v1/admin/metrics` for single-node SQLite fallback (same shape). |
|
||||||
| Retire `cluster-dashboard` | ☐ | Remove package once Admin Health tab ships. Update `defaultBundledPackages`. |
|
| Retire `cluster-dashboard` | ✅ | Remove package once Admin Health tab ships. Update `defaultBundledPackages`. |
|
||||||
| Fix block renderer `requires` | ☐ | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` all have `"requires": ["chat"]`. These are content renderers, not chat features. Remove constraint — they should activate without chat. |
|
| Fix block renderer `requires` | ✅ | `mermaid-renderer`, `katex-renderer`, `csv-table`, `diff-viewer` all have `"requires": ["chat"]`. These are content renderers, not chat features. Remove constraint — they should activate without chat. |
|
||||||
| Health endpoint consolidation | ☐ | `/health` and `/api/v1/health` return near-identical JSON. Merge or clearly differentiate with docs. |
|
| Health endpoint consolidation | ✅ | `/health` and `/api/v1/health` return near-identical JSON. Merge or clearly differentiate with docs. |
|
||||||
|
|
||||||
### v0.6.5 — Renderer Pipeline + Docs Rewrite
|
### v0.6.5 — Renderer Pipeline + Docs Rewrite
|
||||||
|
|
||||||
|
|||||||
@@ -1,100 +0,0 @@
|
|||||||
.cluster-dashboard {
|
|
||||||
max-width: 1200px;
|
|
||||||
margin: 0 auto;
|
|
||||||
padding: 2rem 1.5rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-header {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 1rem;
|
|
||||||
margin-bottom: 1.5rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-header h1 {
|
|
||||||
margin: 0;
|
|
||||||
font-size: 1.5rem;
|
|
||||||
color: var(--text-primary, #111);
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-status {
|
|
||||||
font-size: 0.85rem;
|
|
||||||
color: var(--text-secondary, #666);
|
|
||||||
background: var(--bg-secondary, #f3f4f6);
|
|
||||||
padding: 0.25rem 0.75rem;
|
|
||||||
border-radius: 999px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-grid {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(auto-fill, minmax(320px, 1fr));
|
|
||||||
gap: 1rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-card {
|
|
||||||
background: var(--bg-primary, #fff);
|
|
||||||
border: 1px solid var(--border, #e5e7eb);
|
|
||||||
border-radius: 8px;
|
|
||||||
padding: 1.25rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.card-header {
|
|
||||||
display: flex;
|
|
||||||
justify-content: space-between;
|
|
||||||
align-items: center;
|
|
||||||
margin-bottom: 0.5rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.node-id {
|
|
||||||
font-weight: 600;
|
|
||||||
font-size: 0.9rem;
|
|
||||||
font-family: var(--font-mono, monospace);
|
|
||||||
color: var(--text-primary, #111);
|
|
||||||
}
|
|
||||||
|
|
||||||
.heartbeat-badge {
|
|
||||||
font-size: 0.75rem;
|
|
||||||
color: var(--text-secondary, #666);
|
|
||||||
background: var(--bg-secondary, #f3f4f6);
|
|
||||||
padding: 0.15rem 0.5rem;
|
|
||||||
border-radius: 4px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.node-endpoint {
|
|
||||||
font-size: 0.8rem;
|
|
||||||
color: var(--text-secondary, #666);
|
|
||||||
margin-bottom: 0.75rem;
|
|
||||||
font-family: var(--font-mono, monospace);
|
|
||||||
}
|
|
||||||
|
|
||||||
.card-stats {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: 1fr 1fr 1fr;
|
|
||||||
gap: 0.5rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.stat {
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
gap: 0.15rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.stat-label {
|
|
||||||
font-size: 0.7rem;
|
|
||||||
text-transform: uppercase;
|
|
||||||
letter-spacing: 0.03em;
|
|
||||||
color: var(--text-secondary, #999);
|
|
||||||
}
|
|
||||||
|
|
||||||
.stat-value {
|
|
||||||
font-size: 0.9rem;
|
|
||||||
font-weight: 500;
|
|
||||||
font-family: var(--font-mono, monospace);
|
|
||||||
color: var(--text-primary, #111);
|
|
||||||
}
|
|
||||||
|
|
||||||
.cluster-empty {
|
|
||||||
color: var(--text-secondary, #666);
|
|
||||||
text-align: center;
|
|
||||||
padding: 3rem 1rem;
|
|
||||||
}
|
|
||||||
@@ -1,124 +0,0 @@
|
|||||||
/**
|
|
||||||
* Cluster Dashboard — admin surface showing registered cluster nodes.
|
|
||||||
*
|
|
||||||
* Polls GET /api/v1/admin/cluster every 10s. Renders one card per node
|
|
||||||
* with runtime stats from the JSONB stats column. New keys added to
|
|
||||||
* stats are rendered automatically — no code change required.
|
|
||||||
*/
|
|
||||||
(function () {
|
|
||||||
'use strict';
|
|
||||||
|
|
||||||
var mount = document.getElementById('extension-mount');
|
|
||||||
if (!mount) return;
|
|
||||||
|
|
||||||
var REFRESH_MS = 10000;
|
|
||||||
var timer = null;
|
|
||||||
|
|
||||||
mount.innerHTML =
|
|
||||||
'<div class="cluster-dashboard">' +
|
|
||||||
'<div class="cluster-header">' +
|
|
||||||
'<h1>Cluster</h1>' +
|
|
||||||
'<span class="cluster-status" id="clusterStatus">loading...</span>' +
|
|
||||||
'</div>' +
|
|
||||||
'<div class="cluster-grid" id="clusterGrid"></div>' +
|
|
||||||
'</div>';
|
|
||||||
|
|
||||||
load();
|
|
||||||
timer = setInterval(load, REFRESH_MS);
|
|
||||||
|
|
||||||
// Cleanup on navigation (SPA)
|
|
||||||
window.addEventListener('beforeunload', function () {
|
|
||||||
if (timer) clearInterval(timer);
|
|
||||||
});
|
|
||||||
|
|
||||||
function load() {
|
|
||||||
var api = (window.sw && window.sw.api) || (typeof API !== 'undefined' && API);
|
|
||||||
if (!api) return;
|
|
||||||
var get = api._get || api.get;
|
|
||||||
if (!get) return;
|
|
||||||
get.call(api, '/api/v1/admin/cluster').then(render).catch(function () {
|
|
||||||
// 404 = SQLite (no cluster route registered), or other fetch error
|
|
||||||
render({ data: [] });
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function render(resp) {
|
|
||||||
var nodes = (resp && resp.data) || [];
|
|
||||||
var status = document.getElementById('clusterStatus');
|
|
||||||
var grid = document.getElementById('clusterGrid');
|
|
||||||
|
|
||||||
status.textContent = nodes.length + ' node' + (nodes.length !== 1 ? 's' : '') + ' registered';
|
|
||||||
|
|
||||||
if (nodes.length === 0) {
|
|
||||||
grid.innerHTML = '<p class="cluster-empty">No nodes registered. The cluster registry requires PostgreSQL — SQLite runs single-node only.</p>';
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
grid.innerHTML = nodes.map(nodeCard).join('');
|
|
||||||
}
|
|
||||||
|
|
||||||
function nodeCard(node) {
|
|
||||||
var stats = {};
|
|
||||||
try { stats = typeof node.stats === 'string' ? JSON.parse(node.stats) : (node.stats || {}); } catch (e) { /* ignore */ }
|
|
||||||
|
|
||||||
var uptimeSec = stats.uptime_sec || 0;
|
|
||||||
var uptime = formatDuration(uptimeSec);
|
|
||||||
var heap = formatBytes(stats.heap_alloc || 0);
|
|
||||||
var gcPause = ((stats.gc_pause_ns || 0) / 1e6).toFixed(1) + ' ms';
|
|
||||||
var wsClients = stats.ws_clients != null ? stats.ws_clients : '-';
|
|
||||||
var goroutines = stats.goroutines != null ? stats.goroutines : '-';
|
|
||||||
var gcCycles = stats.gc_cycles != null ? stats.gc_cycles : '-';
|
|
||||||
var heartbeatAge = timeSince(node.heartbeat);
|
|
||||||
|
|
||||||
return (
|
|
||||||
'<div class="cluster-card">' +
|
|
||||||
'<div class="card-header">' +
|
|
||||||
'<span class="node-id">' + esc(node.node_id) + '</span>' +
|
|
||||||
'<span class="heartbeat-badge" title="Last heartbeat">' + heartbeatAge + '</span>' +
|
|
||||||
'</div>' +
|
|
||||||
(node.endpoint ? '<div class="node-endpoint">' + esc(node.endpoint) + '</div>' : '') +
|
|
||||||
'<div class="card-stats">' +
|
|
||||||
stat('Uptime', uptime) +
|
|
||||||
stat('WS Clients', wsClients) +
|
|
||||||
stat('Goroutines', goroutines) +
|
|
||||||
stat('Heap', heap) +
|
|
||||||
stat('GC Pause', gcPause) +
|
|
||||||
stat('GC Cycles', gcCycles) +
|
|
||||||
'</div>' +
|
|
||||||
'</div>'
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
function stat(label, value) {
|
|
||||||
return '<div class="stat"><span class="stat-label">' + label + '</span><span class="stat-value">' + value + '</span></div>';
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatBytes(b) {
|
|
||||||
if (b < 1024) return b + ' B';
|
|
||||||
if (b < 1024 * 1024) return (b / 1024).toFixed(1) + ' KB';
|
|
||||||
return (b / (1024 * 1024)).toFixed(1) + ' MB';
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatDuration(sec) {
|
|
||||||
if (sec < 60) return Math.floor(sec) + 's';
|
|
||||||
if (sec < 3600) return Math.floor(sec / 60) + 'm ' + Math.floor(sec % 60) + 's';
|
|
||||||
var h = Math.floor(sec / 3600);
|
|
||||||
var m = Math.floor((sec % 3600) / 60);
|
|
||||||
return h + 'h ' + m + 'm';
|
|
||||||
}
|
|
||||||
|
|
||||||
function timeSince(iso) {
|
|
||||||
if (!iso) return '-';
|
|
||||||
var ms = Date.now() - new Date(iso).getTime();
|
|
||||||
if (ms < 0) ms = 0;
|
|
||||||
var sec = Math.floor(ms / 1000);
|
|
||||||
if (sec < 60) return sec + 's ago';
|
|
||||||
return Math.floor(sec / 60) + 'm ago';
|
|
||||||
}
|
|
||||||
|
|
||||||
function esc(s) {
|
|
||||||
var d = document.createElement('div');
|
|
||||||
d.appendChild(document.createTextNode(s || ''));
|
|
||||||
return d.innerHTML;
|
|
||||||
}
|
|
||||||
})();
|
|
||||||
@@ -1,13 +0,0 @@
|
|||||||
{
|
|
||||||
"id": "cluster-dashboard",
|
|
||||||
"icon": "🔗",
|
|
||||||
"type": "surface",
|
|
||||||
"title": "Cluster",
|
|
||||||
"route": "/s/cluster-dashboard",
|
|
||||||
"auth": "admin",
|
|
||||||
"layout": "single",
|
|
||||||
"components": [],
|
|
||||||
"hooks": ["surface"],
|
|
||||||
"version": "0.6.0",
|
|
||||||
"description": "Cluster node registry — health, runtime stats, and peer visibility."
|
|
||||||
}
|
|
||||||
@@ -6,7 +6,7 @@
|
|||||||
"tier": "browser",
|
"tier": "browser",
|
||||||
"author": "switchboard",
|
"author": "switchboard",
|
||||||
"description": "Renders ```csv code blocks as sortable, interactive HTML tables",
|
"description": "Renders ```csv code blocks as sortable, interactive HTML tables",
|
||||||
"requires": ["chat"],
|
"requires": [],
|
||||||
"permissions": [],
|
"permissions": [],
|
||||||
"tools": [],
|
"tools": [],
|
||||||
"surfaces": [],
|
"surfaces": [],
|
||||||
|
|||||||
@@ -6,7 +6,7 @@
|
|||||||
"tier": "browser",
|
"tier": "browser",
|
||||||
"author": "switchboard",
|
"author": "switchboard",
|
||||||
"description": "Renders ```diff code blocks with syntax-highlighted additions, deletions, and context lines",
|
"description": "Renders ```diff code blocks with syntax-highlighted additions, deletions, and context lines",
|
||||||
"requires": ["chat"],
|
"requires": [],
|
||||||
"permissions": [],
|
"permissions": [],
|
||||||
"tools": [],
|
"tools": [],
|
||||||
"surfaces": [],
|
"surfaces": [],
|
||||||
|
|||||||
@@ -6,7 +6,7 @@
|
|||||||
"tier": "browser",
|
"tier": "browser",
|
||||||
"author": "switchboard",
|
"author": "switchboard",
|
||||||
"description": "Renders LaTeX math expressions: ```latex blocks and inline $...$ / $$...$$ syntax",
|
"description": "Renders LaTeX math expressions: ```latex blocks and inline $...$ / $$...$$ syntax",
|
||||||
"requires": ["chat"],
|
"requires": [],
|
||||||
"permissions": [],
|
"permissions": [],
|
||||||
"tools": [],
|
"tools": [],
|
||||||
"surfaces": [],
|
"surfaces": [],
|
||||||
|
|||||||
@@ -6,7 +6,7 @@
|
|||||||
"tier": "browser",
|
"tier": "browser",
|
||||||
"author": "switchboard",
|
"author": "switchboard",
|
||||||
"description": "Renders ```mermaid code blocks as interactive SVG diagrams with zoom/pan, SVG/PNG export, and source copy",
|
"description": "Renders ```mermaid code blocks as interactive SVG diagrams with zoom/pan, SVG/PNG export, and source copy",
|
||||||
"requires": ["chat"],
|
"requires": [],
|
||||||
"permissions": [],
|
"permissions": [],
|
||||||
"tools": [],
|
"tools": [],
|
||||||
"surfaces": [],
|
"surfaces": [],
|
||||||
|
|||||||
@@ -34,6 +34,26 @@ type Registry struct {
|
|||||||
startTime time.Time
|
startTime time.Time
|
||||||
stopCh chan struct{}
|
stopCh chan struct{}
|
||||||
wg sync.WaitGroup
|
wg sync.WaitGroup
|
||||||
|
|
||||||
|
// Optional callbacks — set after construction via setters.
|
||||||
|
sandboxStats func() (exec, errors uint64, avgMs float64)
|
||||||
|
triggerFires func() int64
|
||||||
|
extensionCount func() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetSandboxStats registers a callback for sandbox execution counters.
|
||||||
|
func (r *Registry) SetSandboxStats(fn func() (uint64, uint64, float64)) {
|
||||||
|
r.sandboxStats = fn
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetTriggerFireCount registers a callback for trigger fire count.
|
||||||
|
func (r *Registry) SetTriggerFireCount(fn func() int64) {
|
||||||
|
r.triggerFires = fn
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetExtensionCount registers a callback for active extension count.
|
||||||
|
func (r *Registry) SetExtensionCount(fn func() int) {
|
||||||
|
r.extensionCount = fn
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewRegistry creates a cluster registry instance.
|
// NewRegistry creates a cluster registry instance.
|
||||||
@@ -129,13 +149,28 @@ func (r *Registry) collectStats() json.RawMessage {
|
|||||||
runtime.ReadMemStats(&m)
|
runtime.ReadMemStats(&m)
|
||||||
|
|
||||||
stats := map[string]any{
|
stats := map[string]any{
|
||||||
"goroutines": runtime.NumGoroutine(),
|
"goroutines": runtime.NumGoroutine(),
|
||||||
"heap_alloc": m.HeapAlloc,
|
"heap_alloc": m.HeapAlloc,
|
||||||
"heap_sys": m.HeapSys,
|
"heap_sys": m.HeapSys,
|
||||||
"gc_cycles": m.NumGC,
|
"stack_in_use": m.StackInuse,
|
||||||
"gc_pause_ns": m.PauseNs[(m.NumGC+255)%256],
|
"gc_cycles": m.NumGC,
|
||||||
"uptime_sec": time.Since(r.startTime).Seconds(),
|
"gc_pause_ns": m.PauseNs[(m.NumGC+255)%256],
|
||||||
"ws_clients": r.hub.ConnCount(),
|
"gc_cpu_pct": m.GCCPUFraction * 100,
|
||||||
|
"uptime_sec": time.Since(r.startTime).Seconds(),
|
||||||
|
"ws_clients": r.hub.ConnCount(),
|
||||||
|
}
|
||||||
|
|
||||||
|
if r.extensionCount != nil {
|
||||||
|
stats["extensions_loaded"] = r.extensionCount()
|
||||||
|
}
|
||||||
|
if r.sandboxStats != nil {
|
||||||
|
exec, errors, avgMs := r.sandboxStats()
|
||||||
|
stats["starlark_exec_total"] = exec
|
||||||
|
stats["starlark_errors_total"] = errors
|
||||||
|
stats["starlark_avg_duration_ms"] = avgMs
|
||||||
|
}
|
||||||
|
if r.triggerFires != nil {
|
||||||
|
stats["trigger_fires_total"] = r.triggerFires()
|
||||||
}
|
}
|
||||||
|
|
||||||
data, _ := json.Marshal(stats)
|
data, _ := json.Marshal(stats)
|
||||||
|
|||||||
@@ -75,6 +75,46 @@ func TestCollectStats(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCollectStatsFattened(t *testing.T) {
|
||||||
|
hub := &mockHub{count: 3}
|
||||||
|
reg := NewRegistry("test-node", "", nil, hub, RegistryConfig{
|
||||||
|
HeartbeatInterval: 10 * time.Second,
|
||||||
|
StaleThreshold: 30 * time.Second,
|
||||||
|
})
|
||||||
|
reg.SetSandboxStats(func() (uint64, uint64, float64) { return 100, 5, 12.3 })
|
||||||
|
reg.SetTriggerFireCount(func() int64 { return 42 })
|
||||||
|
reg.SetExtensionCount(func() int { return 8 })
|
||||||
|
|
||||||
|
data := reg.collectStats()
|
||||||
|
|
||||||
|
var stats map[string]any
|
||||||
|
if err := json.Unmarshal(data, &stats); err != nil {
|
||||||
|
t.Fatalf("collectStats returned invalid JSON: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// New fattened keys
|
||||||
|
fatKeys := []string{
|
||||||
|
"stack_in_use", "gc_cpu_pct",
|
||||||
|
"extensions_loaded", "starlark_exec_total", "starlark_errors_total",
|
||||||
|
"starlark_avg_duration_ms", "trigger_fires_total",
|
||||||
|
}
|
||||||
|
for _, key := range fatKeys {
|
||||||
|
if _, ok := stats[key]; !ok {
|
||||||
|
t.Errorf("missing fattened stats key: %s", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if v := stats["extensions_loaded"].(float64); int(v) != 8 {
|
||||||
|
t.Errorf("extensions_loaded = %v, want 8", v)
|
||||||
|
}
|
||||||
|
if v := stats["starlark_exec_total"].(float64); int(v) != 100 {
|
||||||
|
t.Errorf("starlark_exec_total = %v, want 100", v)
|
||||||
|
}
|
||||||
|
if v := stats["trigger_fires_total"].(float64); int(v) != 42 {
|
||||||
|
t.Errorf("trigger_fires_total = %v, want 42", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRegistryStartStop(t *testing.T) {
|
func TestRegistryStartStop(t *testing.T) {
|
||||||
ms := &mockClusterStore{heartbeatRows: 1}
|
ms := &mockClusterStore{heartbeatRows: 1}
|
||||||
hub := &mockHub{count: 0}
|
hub := &mockHub{count: 0}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package events
|
|||||||
import (
|
import (
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -14,6 +15,8 @@ type Bus struct {
|
|||||||
subs map[string][]*subscription
|
subs map[string][]*subscription
|
||||||
seq uint64 // subscription ID counter
|
seq uint64 // subscription ID counter
|
||||||
broadcastHook func(Event) // called after Publish for cross-pod fan-out; nil-safe
|
broadcastHook func(Event) // called after Publish for cross-pod fan-out; nil-safe
|
||||||
|
publishCount atomic.Int64
|
||||||
|
deliverCount atomic.Int64
|
||||||
}
|
}
|
||||||
|
|
||||||
type subscription struct {
|
type subscription struct {
|
||||||
@@ -86,6 +89,8 @@ func (b *Bus) Publish(event Event) {
|
|||||||
// the broadcastHook. Used by the Postgres listener to re-publish remote
|
// the broadcastHook. Used by the Postgres listener to re-publish remote
|
||||||
// events without causing an infinite re-broadcast loop.
|
// events without causing an infinite re-broadcast loop.
|
||||||
func (b *Bus) publishLocal(event Event) {
|
func (b *Bus) publishLocal(event Event) {
|
||||||
|
b.publishCount.Add(1)
|
||||||
|
|
||||||
b.mu.RLock()
|
b.mu.RLock()
|
||||||
var matched []Handler
|
var matched []Handler
|
||||||
for pattern, subs := range b.subs {
|
for pattern, subs := range b.subs {
|
||||||
@@ -98,6 +103,7 @@ func (b *Bus) publishLocal(event Event) {
|
|||||||
b.mu.RUnlock()
|
b.mu.RUnlock()
|
||||||
|
|
||||||
for _, h := range matched {
|
for _, h := range matched {
|
||||||
|
b.deliverCount.Add(1)
|
||||||
h(event)
|
h(event)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -126,6 +132,12 @@ func (b *Bus) PublishAsync(event Event) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PublishCount returns the cumulative number of events published.
|
||||||
|
func (b *Bus) PublishCount() int64 { return b.publishCount.Load() }
|
||||||
|
|
||||||
|
// DeliverCount returns the cumulative number of subscriber deliveries.
|
||||||
|
func (b *Bus) DeliverCount() int64 { return b.deliverCount.Load() }
|
||||||
|
|
||||||
// match checks if a concrete label matches a subscription pattern.
|
// match checks if a concrete label matches a subscription pattern.
|
||||||
//
|
//
|
||||||
// "chat.message.abc" matches "chat.message.abc" (exact)
|
// "chat.message.abc" matches "chat.message.abc" (exact)
|
||||||
|
|||||||
24
server/handlers/admin_metrics.go
Normal file
24
server/handlers/admin_metrics.go
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"github.com/gin-gonic/gin"
|
||||||
|
|
||||||
|
"switchboard-core/metrics"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MetricsHandler serves the admin metrics endpoint.
|
||||||
|
type MetricsHandler struct {
|
||||||
|
collector *metrics.Collector
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewMetricsHandler creates a MetricsHandler.
|
||||||
|
func NewMetricsHandler(c *metrics.Collector) *MetricsHandler {
|
||||||
|
return &MetricsHandler{collector: c}
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetMetrics returns a full metrics snapshot.
|
||||||
|
// GET /api/v1/admin/metrics
|
||||||
|
func (h *MetricsHandler) GetMetrics(c *gin.Context) {
|
||||||
|
snap := h.collector.Collect(c.Request.Context())
|
||||||
|
c.JSON(200, snap)
|
||||||
|
}
|
||||||
179
server/handlers/admin_metrics_test.go
Normal file
179
server/handlers/admin_metrics_test.go
Normal file
@@ -0,0 +1,179 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gin-gonic/gin"
|
||||||
|
|
||||||
|
"switchboard-core/metrics"
|
||||||
|
"switchboard-core/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// mockHub implements metrics.ConnCounter for tests.
|
||||||
|
type mockHub struct{ count int }
|
||||||
|
|
||||||
|
func (m *mockHub) ConnCount() int { return m.count }
|
||||||
|
|
||||||
|
// mockBus implements metrics.BusCounter for tests.
|
||||||
|
type mockBus struct{ pub, del int64 }
|
||||||
|
|
||||||
|
func (m *mockBus) PublishCount() int64 { return m.pub }
|
||||||
|
func (m *mockBus) DeliverCount() int64 { return m.del }
|
||||||
|
|
||||||
|
func TestMetrics_SQLiteShape(t *testing.T) {
|
||||||
|
gin.SetMode(gin.TestMode)
|
||||||
|
|
||||||
|
collector := metrics.NewCollector(
|
||||||
|
nil, // no DB
|
||||||
|
&mockHub{count: 5},
|
||||||
|
&mockBus{pub: 10, del: 20},
|
||||||
|
store.Stores{}, // no cluster store
|
||||||
|
func() (uint64, uint64, float64) { return 42, 3, 12.5 },
|
||||||
|
func() int64 { return 7 },
|
||||||
|
time.Now().Add(-10*time.Minute),
|
||||||
|
)
|
||||||
|
|
||||||
|
h := NewMetricsHandler(collector)
|
||||||
|
r := gin.New()
|
||||||
|
r.GET("/api/v1/admin/metrics", h.GetMetrics)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/admin/metrics", nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("status = %d, want 200", w.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
var snap metrics.Snapshot
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &snap); err != nil {
|
||||||
|
t.Fatalf("unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Runtime
|
||||||
|
if snap.Runtime.WSClients != 5 {
|
||||||
|
t.Errorf("ws_clients = %d, want 5", snap.Runtime.WSClients)
|
||||||
|
}
|
||||||
|
if snap.Runtime.UptimeSec < 600 {
|
||||||
|
t.Errorf("uptime_sec = %f, want >= 600", snap.Runtime.UptimeSec)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cluster should be nil (SQLite)
|
||||||
|
if snap.Cluster != nil {
|
||||||
|
t.Errorf("cluster should be nil for SQLite, got %+v", snap.Cluster)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extensions
|
||||||
|
if snap.Extensions.StarlarkExecTotal != 42 {
|
||||||
|
t.Errorf("starlark_exec_total = %d, want 42", snap.Extensions.StarlarkExecTotal)
|
||||||
|
}
|
||||||
|
if snap.Extensions.StarlarkErrorsTotal != 3 {
|
||||||
|
t.Errorf("starlark_errors_total = %d, want 3", snap.Extensions.StarlarkErrorsTotal)
|
||||||
|
}
|
||||||
|
if snap.Extensions.StarlarkAvgDuration != 12.5 {
|
||||||
|
t.Errorf("starlark_avg_duration_ms = %f, want 12.5", snap.Extensions.StarlarkAvgDuration)
|
||||||
|
}
|
||||||
|
if snap.Extensions.TriggerFiresTotal != 7 {
|
||||||
|
t.Errorf("trigger_fires_total = %d, want 7", snap.Extensions.TriggerFiresTotal)
|
||||||
|
}
|
||||||
|
if snap.Extensions.EventBusPublished != 10 {
|
||||||
|
t.Errorf("event_bus_published = %d, want 10", snap.Extensions.EventBusPublished)
|
||||||
|
}
|
||||||
|
if snap.Extensions.EventBusDelivered != 20 {
|
||||||
|
t.Errorf("event_bus_delivered = %d, want 20", snap.Extensions.EventBusDelivered)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMetrics_WithCluster(t *testing.T) {
|
||||||
|
gin.SetMode(gin.TestMode)
|
||||||
|
|
||||||
|
mock := &mockClusterStore{
|
||||||
|
nodes: []store.ClusterNode{
|
||||||
|
{
|
||||||
|
NodeID: "node-1",
|
||||||
|
Endpoint: "http://node-1:8080",
|
||||||
|
Heartbeat: time.Now(),
|
||||||
|
Stats: json.RawMessage(`{"uptime_sec":120,"ws_clients":3}`),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
collector := metrics.NewCollector(
|
||||||
|
nil,
|
||||||
|
&mockHub{count: 3},
|
||||||
|
&mockBus{},
|
||||||
|
store.Stores{Cluster: mock},
|
||||||
|
func() (uint64, uint64, float64) { return 0, 0, 0 },
|
||||||
|
func() int64 { return 0 },
|
||||||
|
time.Now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
h := NewMetricsHandler(collector)
|
||||||
|
r := gin.New()
|
||||||
|
r.GET("/api/v1/admin/metrics", h.GetMetrics)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/admin/metrics", nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("status = %d, want 200", w.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
var snap metrics.Snapshot
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &snap); err != nil {
|
||||||
|
t.Fatalf("unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if snap.Cluster == nil {
|
||||||
|
t.Fatal("cluster should not be nil with cluster store")
|
||||||
|
}
|
||||||
|
if snap.Cluster.Size != 1 {
|
||||||
|
t.Errorf("cluster.size = %d, want 1", snap.Cluster.Size)
|
||||||
|
}
|
||||||
|
if snap.Cluster.Nodes[0].NodeID != "node-1" {
|
||||||
|
t.Errorf("node_id = %q, want %q", snap.Cluster.Nodes[0].NodeID, "node-1")
|
||||||
|
}
|
||||||
|
if snap.Cluster.Nodes[0].UptimeSec != 120 {
|
||||||
|
t.Errorf("uptime_sec = %f, want 120", snap.Cluster.Nodes[0].UptimeSec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMetrics_ExtensionCounters(t *testing.T) {
|
||||||
|
gin.SetMode(gin.TestMode)
|
||||||
|
|
||||||
|
collector := metrics.NewCollector(
|
||||||
|
nil,
|
||||||
|
&mockHub{},
|
||||||
|
&mockBus{pub: 100, del: 500},
|
||||||
|
store.Stores{},
|
||||||
|
func() (uint64, uint64, float64) { return 1000, 50, 8.3 },
|
||||||
|
func() int64 { return 25 },
|
||||||
|
time.Now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
h := NewMetricsHandler(collector)
|
||||||
|
r := gin.New()
|
||||||
|
r.GET("/api/v1/admin/metrics", h.GetMetrics)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/admin/metrics", nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
var snap metrics.Snapshot
|
||||||
|
_ = json.Unmarshal(w.Body.Bytes(), &snap)
|
||||||
|
|
||||||
|
if snap.Extensions.StarlarkExecTotal != 1000 {
|
||||||
|
t.Errorf("exec total = %d, want 1000", snap.Extensions.StarlarkExecTotal)
|
||||||
|
}
|
||||||
|
if snap.Extensions.EventBusPublished != 100 {
|
||||||
|
t.Errorf("bus published = %d, want 100", snap.Extensions.EventBusPublished)
|
||||||
|
}
|
||||||
|
if snap.Extensions.EventBusDelivered != 500 {
|
||||||
|
t.Errorf("bus delivered = %d, want 500", snap.Extensions.EventBusDelivered)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -56,6 +56,7 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Server startup ──────────────────────
|
// ── Server startup ──────────────────────
|
||||||
|
startTime := time.Now()
|
||||||
cfg := config.Load()
|
cfg := config.Load()
|
||||||
|
|
||||||
// Structured logging — must be first so all subsequent
|
// Structured logging — must be first so all subsequent
|
||||||
@@ -213,6 +214,24 @@ func main() {
|
|||||||
HeartbeatInterval: cfg.ClusterHeartbeatInterval,
|
HeartbeatInterval: cfg.ClusterHeartbeatInterval,
|
||||||
StaleThreshold: cfg.ClusterStaleThreshold,
|
StaleThreshold: cfg.ClusterStaleThreshold,
|
||||||
})
|
})
|
||||||
|
clusterReg.SetSandboxStats(sandbox.SandboxStats)
|
||||||
|
clusterReg.SetTriggerFireCount(triggerEngine.FireCount)
|
||||||
|
clusterReg.SetExtensionCount(func() int {
|
||||||
|
if stores.Packages == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
pkgs, err := stores.Packages.List(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
count := 0
|
||||||
|
for _, p := range pkgs {
|
||||||
|
if p.Status == "active" {
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
})
|
||||||
if err := clusterReg.Start(); err != nil {
|
if err := clusterReg.Start(); err != nil {
|
||||||
log.Printf("⚠ Cluster registry failed to start: %v", err)
|
log.Printf("⚠ Cluster registry failed to start: %v", err)
|
||||||
clusterReg = nil
|
clusterReg = nil
|
||||||
@@ -255,7 +274,7 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Health check (k8s probes hit this directly)
|
// Health check (k8s probes hit this directly)
|
||||||
base.GET("/health", func(c *gin.Context) {
|
buildHealthResponse := func() gin.H {
|
||||||
info := gin.H{
|
info := gin.H{
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"version": Version,
|
"version": Version,
|
||||||
@@ -263,8 +282,15 @@ func main() {
|
|||||||
"database_name": database.Name(),
|
"database_name": database.Name(),
|
||||||
"schema_version": database.SchemaVersion(),
|
"schema_version": database.SchemaVersion(),
|
||||||
}
|
}
|
||||||
|
if database.IsConnected() {
|
||||||
|
info["registration_enabled"] = handlers.IsRegistrationEnabled(stores)
|
||||||
|
}
|
||||||
appendClusterHealth(info, clusterReg, stores)
|
appendClusterHealth(info, clusterReg, stores)
|
||||||
c.JSON(200, info)
|
return info
|
||||||
|
}
|
||||||
|
|
||||||
|
base.GET("/health", func(c *gin.Context) {
|
||||||
|
c.JSON(200, buildHealthResponse())
|
||||||
})
|
})
|
||||||
|
|
||||||
// Liveness: process is alive and serving (no dependency checks).
|
// Liveness: process is alive and serving (no dependency checks).
|
||||||
@@ -352,20 +378,9 @@ func main() {
|
|||||||
|
|
||||||
api := base.Group("/api/v1")
|
api := base.Group("/api/v1")
|
||||||
{
|
{
|
||||||
// Health (routable through ingress)
|
// Health (routable through ingress — same shape as /health)
|
||||||
api.GET("/health", func(c *gin.Context) {
|
api.GET("/health", func(c *gin.Context) {
|
||||||
info := gin.H{
|
c.JSON(200, buildHealthResponse())
|
||||||
"status": "ok",
|
|
||||||
"version": Version,
|
|
||||||
"schema_version": database.SchemaVersion(),
|
|
||||||
"database": database.IsConnected(),
|
|
||||||
"database_name": database.Name(),
|
|
||||||
}
|
|
||||||
if database.IsConnected() {
|
|
||||||
info["registration_enabled"] = handlers.IsRegistrationEnabled(stores)
|
|
||||||
}
|
|
||||||
appendClusterHealth(info, clusterReg, stores)
|
|
||||||
c.JSON(200, info)
|
|
||||||
})
|
})
|
||||||
|
|
||||||
authGroup := api.Group("/auth")
|
authGroup := api.Group("/auth")
|
||||||
@@ -794,6 +809,16 @@ func main() {
|
|||||||
admin.GET("/cluster", clusterH.ListNodes)
|
admin.GET("/cluster", clusterH.ListNodes)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Metrics ─────────────────
|
||||||
|
metricsCollector := metrics.NewCollector(
|
||||||
|
database.DB, hub, bus, stores,
|
||||||
|
sandbox.SandboxStats,
|
||||||
|
triggerEngine.FireCount,
|
||||||
|
startTime,
|
||||||
|
)
|
||||||
|
metricsH := handlers.NewMetricsHandler(metricsCollector)
|
||||||
|
admin.GET("/metrics", metricsH.GetMetrics)
|
||||||
|
|
||||||
// ── Backup/Restore ─────────
|
// ── Backup/Restore ─────────
|
||||||
backupH := handlers.NewBackupHandler(stores, packagesDir, cfg.StoragePath)
|
backupH := handlers.NewBackupHandler(stores, packagesDir, cfg.StoragePath)
|
||||||
admin.POST("/backup", backupH.CreateBackup)
|
admin.POST("/backup", backupH.CreateBackup)
|
||||||
|
|||||||
253
server/metrics/collector.go
Normal file
253
server/metrics/collector.go
Normal file
@@ -0,0 +1,253 @@
|
|||||||
|
// Package metrics — collector.go
|
||||||
|
//
|
||||||
|
// On-demand metrics collector for the admin /api/v1/admin/metrics endpoint.
|
||||||
|
// Gathers runtime, database, cluster, and extension stats into a single JSON snapshot.
|
||||||
|
package metrics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"runtime"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"switchboard-core/database"
|
||||||
|
"switchboard-core/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Snapshot is the top-level JSON response from GET /api/v1/admin/metrics.
|
||||||
|
type Snapshot struct {
|
||||||
|
Runtime RuntimeMetrics `json:"runtime"`
|
||||||
|
DB DBMetrics `json:"db"`
|
||||||
|
Cluster *ClusterMetrics `json:"cluster,omitempty"`
|
||||||
|
Extensions ExtensionMetrics `json:"extensions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type RuntimeMetrics struct {
|
||||||
|
Goroutines int `json:"goroutines"`
|
||||||
|
HeapAlloc uint64 `json:"heap_alloc"`
|
||||||
|
HeapSys uint64 `json:"heap_sys"`
|
||||||
|
StackInUse uint64 `json:"stack_in_use"`
|
||||||
|
GCCycles uint32 `json:"gc_cycles"`
|
||||||
|
GCPauseNs uint64 `json:"gc_pause_ns"`
|
||||||
|
GCCPUPercent float64 `json:"gc_cpu_pct"`
|
||||||
|
UptimeSec float64 `json:"uptime_sec"`
|
||||||
|
WSClients int `json:"ws_clients"`
|
||||||
|
ExtensionsLoaded int `json:"extensions_loaded"`
|
||||||
|
OpenFDs int `json:"open_fds"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type DBMetrics struct {
|
||||||
|
LatencyMs float64 `json:"latency_ms"`
|
||||||
|
PoolActive int `json:"pool_active"`
|
||||||
|
PoolIdle int `json:"pool_idle"`
|
||||||
|
PoolMax int `json:"pool_max"`
|
||||||
|
WaitCount int64 `json:"wait_count"`
|
||||||
|
WaitDuration float64 `json:"wait_duration_ms"`
|
||||||
|
// PG-only fields (zero/omitted on SQLite)
|
||||||
|
DeadTuples *int64 `json:"dead_tuples,omitempty"`
|
||||||
|
ActiveBackends *int `json:"active_backends,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ClusterMetrics struct {
|
||||||
|
Size int `json:"size"`
|
||||||
|
Nodes []ClusterNode `json:"nodes"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ClusterNode struct {
|
||||||
|
NodeID string `json:"node_id"`
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
UptimeSec float64 `json:"uptime_sec"`
|
||||||
|
HeartbeatAge int64 `json:"heartbeat_age_ms"`
|
||||||
|
Stats json.RawMessage `json:"stats"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ExtensionMetrics struct {
|
||||||
|
StarlarkExecTotal uint64 `json:"starlark_exec_total"`
|
||||||
|
StarlarkErrorsTotal uint64 `json:"starlark_errors_total"`
|
||||||
|
StarlarkAvgDuration float64 `json:"starlark_avg_duration_ms"`
|
||||||
|
TriggerFiresTotal int64 `json:"trigger_fires_total"`
|
||||||
|
EventBusPublished int64 `json:"event_bus_published"`
|
||||||
|
EventBusDelivered int64 `json:"event_bus_delivered"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ConnCounter provides WebSocket connection count (satisfied by events.Hub).
|
||||||
|
type ConnCounter interface {
|
||||||
|
ConnCount() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// BusCounter provides publish/deliver counts (satisfied by events.Bus).
|
||||||
|
type BusCounter interface {
|
||||||
|
PublishCount() int64
|
||||||
|
DeliverCount() int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// SandboxStatsFunc returns cumulative sandbox execution counters.
|
||||||
|
type SandboxStatsFunc func() (execCount, errorCount uint64, avgDurationMs float64)
|
||||||
|
|
||||||
|
// TriggerFireCountFunc returns cumulative trigger fire count.
|
||||||
|
type TriggerFireCountFunc func() int64
|
||||||
|
|
||||||
|
// Collector gathers metrics on demand for the admin endpoint.
|
||||||
|
type Collector struct {
|
||||||
|
db *sql.DB
|
||||||
|
hub ConnCounter
|
||||||
|
bus BusCounter
|
||||||
|
stores store.Stores
|
||||||
|
sandboxStats SandboxStatsFunc
|
||||||
|
triggerFireCount TriggerFireCountFunc
|
||||||
|
startTime time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewCollector creates a metrics collector with all required dependencies.
|
||||||
|
func NewCollector(db *sql.DB, hub ConnCounter, bus BusCounter, stores store.Stores, sandboxFn SandboxStatsFunc, triggerFn TriggerFireCountFunc, startTime time.Time) *Collector {
|
||||||
|
return &Collector{
|
||||||
|
db: db,
|
||||||
|
hub: hub,
|
||||||
|
bus: bus,
|
||||||
|
stores: stores,
|
||||||
|
sandboxStats: sandboxFn,
|
||||||
|
triggerFireCount: triggerFn,
|
||||||
|
startTime: startTime,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collect gathers all metrics synchronously and returns a snapshot.
|
||||||
|
func (c *Collector) Collect(ctx context.Context) *Snapshot {
|
||||||
|
snap := &Snapshot{}
|
||||||
|
snap.Runtime = c.collectRuntime()
|
||||||
|
snap.DB = c.collectDB(ctx)
|
||||||
|
snap.Cluster = c.collectCluster(ctx)
|
||||||
|
snap.Extensions = c.collectExtensions()
|
||||||
|
return snap
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) collectRuntime() RuntimeMetrics {
|
||||||
|
var m runtime.MemStats
|
||||||
|
runtime.ReadMemStats(&m)
|
||||||
|
|
||||||
|
extCount := 0
|
||||||
|
if c.stores.Packages != nil {
|
||||||
|
if pkgs, err := c.stores.Packages.List(context.Background()); err == nil {
|
||||||
|
for _, p := range pkgs {
|
||||||
|
if p.Status == "active" {
|
||||||
|
extCount++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return RuntimeMetrics{
|
||||||
|
Goroutines: runtime.NumGoroutine(),
|
||||||
|
HeapAlloc: m.HeapAlloc,
|
||||||
|
HeapSys: m.HeapSys,
|
||||||
|
StackInUse: m.StackInuse,
|
||||||
|
GCCycles: m.NumGC,
|
||||||
|
GCPauseNs: m.PauseNs[(m.NumGC+255)%256],
|
||||||
|
GCCPUPercent: m.GCCPUFraction * 100,
|
||||||
|
UptimeSec: time.Since(c.startTime).Seconds(),
|
||||||
|
WSClients: c.hub.ConnCount(),
|
||||||
|
ExtensionsLoaded: extCount,
|
||||||
|
OpenFDs: countOpenFDs(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) collectDB(ctx context.Context) DBMetrics {
|
||||||
|
dm := DBMetrics{}
|
||||||
|
if c.db == nil {
|
||||||
|
return dm
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pool stats
|
||||||
|
stats := c.db.Stats()
|
||||||
|
dm.PoolActive = stats.InUse
|
||||||
|
dm.PoolIdle = stats.Idle
|
||||||
|
dm.PoolMax = stats.MaxOpenConnections
|
||||||
|
dm.WaitCount = stats.WaitCount
|
||||||
|
dm.WaitDuration = float64(stats.WaitDuration.Milliseconds())
|
||||||
|
|
||||||
|
// Latency probe
|
||||||
|
probeCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
start := time.Now()
|
||||||
|
if err := c.db.PingContext(probeCtx); err == nil {
|
||||||
|
dm.LatencyMs = float64(time.Since(start).Microseconds()) / 1000.0
|
||||||
|
}
|
||||||
|
|
||||||
|
// PG-only stats
|
||||||
|
if database.IsPostgres() {
|
||||||
|
var deadTuples int64
|
||||||
|
if err := c.db.QueryRowContext(ctx, `SELECT COALESCE(SUM(n_dead_tup), 0) FROM pg_stat_user_tables`).Scan(&deadTuples); err == nil {
|
||||||
|
dm.DeadTuples = &deadTuples
|
||||||
|
}
|
||||||
|
var activeBackends int
|
||||||
|
if err := c.db.QueryRowContext(ctx, `SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()`).Scan(&activeBackends); err == nil {
|
||||||
|
dm.ActiveBackends = &activeBackends
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return dm
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) collectCluster(ctx context.Context) *ClusterMetrics {
|
||||||
|
if c.stores.Cluster == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
nodes, err := c.stores.Cluster.ListNodes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if len(nodes) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cm := &ClusterMetrics{
|
||||||
|
Size: len(nodes),
|
||||||
|
Nodes: make([]ClusterNode, len(nodes)),
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
for i, n := range nodes {
|
||||||
|
// Extract uptime from stats JSONB
|
||||||
|
var uptimeSec float64
|
||||||
|
var statsMap map[string]any
|
||||||
|
if json.Unmarshal(n.Stats, &statsMap) == nil {
|
||||||
|
if u, ok := statsMap["uptime_sec"].(float64); ok {
|
||||||
|
uptimeSec = u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cm.Nodes[i] = ClusterNode{
|
||||||
|
NodeID: n.NodeID,
|
||||||
|
Endpoint: n.Endpoint,
|
||||||
|
UptimeSec: uptimeSec,
|
||||||
|
HeartbeatAge: now.Sub(n.Heartbeat).Milliseconds(),
|
||||||
|
Stats: n.Stats,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cm
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) collectExtensions() ExtensionMetrics {
|
||||||
|
em := ExtensionMetrics{}
|
||||||
|
if c.sandboxStats != nil {
|
||||||
|
em.StarlarkExecTotal, em.StarlarkErrorsTotal, em.StarlarkAvgDuration = c.sandboxStats()
|
||||||
|
}
|
||||||
|
if c.triggerFireCount != nil {
|
||||||
|
em.TriggerFiresTotal = c.triggerFireCount()
|
||||||
|
}
|
||||||
|
if c.bus != nil {
|
||||||
|
em.EventBusPublished = c.bus.PublishCount()
|
||||||
|
em.EventBusDelivered = c.bus.DeliverCount()
|
||||||
|
}
|
||||||
|
return em
|
||||||
|
}
|
||||||
|
|
||||||
|
// countOpenFDs counts open file descriptors via /proc/self/fd (Linux only).
|
||||||
|
func countOpenFDs() int {
|
||||||
|
entries, err := os.ReadDir("/proc/self/fd")
|
||||||
|
if err != nil {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
return len(entries)
|
||||||
|
}
|
||||||
@@ -23,16 +23,37 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
"go.starlark.net/starlark"
|
"go.starlark.net/starlark"
|
||||||
|
|
||||||
starlarkjson "go.starlark.net/lib/json"
|
starlarkjson "go.starlark.net/lib/json"
|
||||||
|
|
||||||
"switchboard-core/events"
|
"switchboard-core/events"
|
||||||
|
"switchboard-core/metrics"
|
||||||
"switchboard-core/models"
|
"switchboard-core/models"
|
||||||
"switchboard-core/store"
|
"switchboard-core/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// sandboxStats tracks cumulative execution counters for the admin metrics endpoint.
|
||||||
|
var sandboxStats struct {
|
||||||
|
execCount atomic.Int64
|
||||||
|
errorCount atomic.Int64
|
||||||
|
totalDurationNs atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// SandboxStats returns cumulative execution counters.
|
||||||
|
func SandboxStats() (execCount, errorCount uint64, avgDurationMs float64) {
|
||||||
|
exec := sandboxStats.execCount.Load()
|
||||||
|
errs := sandboxStats.errorCount.Load()
|
||||||
|
totalNs := sandboxStats.totalDurationNs.Load()
|
||||||
|
if exec > 0 {
|
||||||
|
avgDurationMs = float64(totalNs) / float64(exec) / 1e6
|
||||||
|
}
|
||||||
|
return uint64(exec), uint64(errs), avgDurationMs
|
||||||
|
}
|
||||||
|
|
||||||
// RunContext carries per-invocation state that modules need but which
|
// RunContext carries per-invocation state that modules need but which
|
||||||
// varies per caller (API route vs filter vs task). Nil is safe — modules
|
// varies per caller (API route vs filter vs task). Nil is safe — modules
|
||||||
// that need RunContext fields gracefully degrade.
|
// that need RunContext fields gracefully degrade.
|
||||||
@@ -142,7 +163,20 @@ func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration
|
|||||||
|
|
||||||
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
|
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
|
||||||
|
|
||||||
return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
|
start := time.Now()
|
||||||
|
result, err := r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
|
||||||
|
duration := time.Since(start)
|
||||||
|
|
||||||
|
sandboxStats.execCount.Add(1)
|
||||||
|
sandboxStats.totalDurationNs.Add(int64(duration))
|
||||||
|
status := "success"
|
||||||
|
if err != nil {
|
||||||
|
sandboxStats.errorCount.Add(1)
|
||||||
|
status = "error"
|
||||||
|
}
|
||||||
|
metrics.SandboxExecutionsTotal.WithLabelValues("exec", status).Inc()
|
||||||
|
|
||||||
|
return result, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// loadScript reads the entry point script from disk (primary path)
|
// loadScript reads the entry point script from disk (primary path)
|
||||||
@@ -266,8 +300,20 @@ func (r *Runner) CallEntryPoint(ctx context.Context, pkg *store.PackageRegistrat
|
|||||||
return nil, result.Output, fmt.Errorf("package %q: %s is not callable", pkg.ID, entryPoint)
|
return nil, result.Output, fmt.Errorf("package %q: %s is not callable", pkg.ID, entryPoint)
|
||||||
}
|
}
|
||||||
|
|
||||||
val, callOutput, err := r.sandbox.Call(ctx, callable, args, kwargs)
|
start := time.Now()
|
||||||
return val, result.Output + callOutput, err
|
val, callOutput, callErr := r.sandbox.Call(ctx, callable, args, kwargs)
|
||||||
|
duration := time.Since(start)
|
||||||
|
|
||||||
|
sandboxStats.execCount.Add(1)
|
||||||
|
sandboxStats.totalDurationNs.Add(int64(duration))
|
||||||
|
status := "success"
|
||||||
|
if callErr != nil {
|
||||||
|
sandboxStats.errorCount.Add(1)
|
||||||
|
status = "error"
|
||||||
|
}
|
||||||
|
metrics.SandboxExecutionsTotal.WithLabelValues(entryPoint, status).Inc()
|
||||||
|
|
||||||
|
return val, result.Output + callOutput, callErr
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildModules assembles the module map based on granted permissions.
|
// buildModules assembles the module map based on granted permissions.
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"log"
|
"log"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/robfig/cron/v3"
|
"github.com/robfig/cron/v3"
|
||||||
@@ -30,10 +31,15 @@ type Engine struct {
|
|||||||
unsubs map[string]func() // trigger_id → bus unsubscribe
|
unsubs map[string]func() // trigger_id → bus unsubscribe
|
||||||
cronIDs map[string]cron.EntryID // scheduled_task_id → cron entry
|
cronIDs map[string]cron.EntryID // scheduled_task_id → cron entry
|
||||||
|
|
||||||
|
fireCount atomic.Int64 // cumulative trigger fires
|
||||||
|
|
||||||
ctx context.Context
|
ctx context.Context
|
||||||
cancel context.CancelFunc
|
cancel context.CancelFunc
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// FireCount returns the cumulative number of trigger fires.
|
||||||
|
func (e *Engine) FireCount() int64 { return e.fireCount.Load() }
|
||||||
|
|
||||||
// New creates a trigger engine. Call Start() to begin processing.
|
// New creates a trigger engine. Call Start() to begin processing.
|
||||||
func New(stores store.Stores, runner *sandbox.Runner, bus *events.Bus) *Engine {
|
func New(stores store.Stores, runner *sandbox.Runner, bus *events.Bus) *Engine {
|
||||||
return &Engine{
|
return &Engine{
|
||||||
@@ -201,8 +207,9 @@ func (e *Engine) logExecution(triggerID, scheduledTaskID string, firedAt time.Ti
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// publishEvent emits a trigger lifecycle event on the bus.
|
// publishEvent emits a trigger lifecycle event on the bus and increments the fire counter.
|
||||||
func (e *Engine) publishEvent(label string, triggerID string) {
|
func (e *Engine) publishEvent(label string, triggerID string) {
|
||||||
|
e.fireCount.Add(1)
|
||||||
if e.bus != nil {
|
if e.bus != nil {
|
||||||
payload, _ := json.Marshal(map[string]any{"trigger_id": triggerID})
|
payload, _ := json.Marshal(map[string]any{"trigger_id": triggerID})
|
||||||
e.bus.Publish(events.Event{
|
e.bus.Publish(events.Event{
|
||||||
|
|||||||
@@ -163,6 +163,7 @@ export function createDomains(restClient) {
|
|||||||
// ── 18. Admin ──────────────────────────
|
// ── 18. Admin ──────────────────────────
|
||||||
admin: {
|
admin: {
|
||||||
stats: () => rc.get('/api/v1/admin/stats'),
|
stats: () => rc.get('/api/v1/admin/stats'),
|
||||||
|
metrics: () => rc.get('/api/v1/admin/metrics'),
|
||||||
|
|
||||||
users: {
|
users: {
|
||||||
list: (opts) => rc.get('/api/v1/admin/users' + _qs(opts)),
|
list: (opts) => rc.get('/api/v1/admin/users' + _qs(opts)),
|
||||||
|
|||||||
206
src/js/sw/surfaces/admin/health.js
Normal file
206
src/js/sw/surfaces/admin/health.js
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
/**
|
||||||
|
* Admin > Monitoring > Health
|
||||||
|
*
|
||||||
|
* Polls GET /api/v1/admin/metrics and displays runtime, DB, cluster,
|
||||||
|
* and extension metrics. Cluster panel only shows on PG multi-node.
|
||||||
|
*/
|
||||||
|
const { html } = window;
|
||||||
|
const { useState, useEffect, useRef, useCallback } = hooks;
|
||||||
|
|
||||||
|
// ── Formatting helpers ──────────────────────────
|
||||||
|
|
||||||
|
function formatBytes(b) {
|
||||||
|
if (b == null) return '\u2014';
|
||||||
|
if (b < 1024) return b + ' B';
|
||||||
|
if (b < 1024 * 1024) return (b / 1024).toFixed(1) + ' KB';
|
||||||
|
if (b < 1024 * 1024 * 1024) return (b / (1024 * 1024)).toFixed(1) + ' MB';
|
||||||
|
return (b / (1024 * 1024 * 1024)).toFixed(2) + ' GB';
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatDuration(sec) {
|
||||||
|
if (sec == null) return '\u2014';
|
||||||
|
if (sec < 60) return Math.floor(sec) + 's';
|
||||||
|
if (sec < 3600) return Math.floor(sec / 60) + 'm ' + Math.floor(sec % 60) + 's';
|
||||||
|
const h = Math.floor(sec / 3600);
|
||||||
|
const m = Math.floor((sec % 3600) / 60);
|
||||||
|
if (h < 24) return h + 'h ' + m + 'm';
|
||||||
|
const d = Math.floor(h / 24);
|
||||||
|
return d + 'd ' + (h % 24) + 'h';
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatMs(ms) {
|
||||||
|
if (ms == null) return '\u2014';
|
||||||
|
if (ms < 1) return '<1 ms';
|
||||||
|
if (ms < 1000) return ms.toFixed(1) + ' ms';
|
||||||
|
return (ms / 1000).toFixed(2) + ' s';
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatPct(v) {
|
||||||
|
if (v == null) return '\u2014';
|
||||||
|
return v.toFixed(2) + '%';
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatNum(v) {
|
||||||
|
if (v == null) return '\u2014';
|
||||||
|
if (typeof v === 'number') return v.toLocaleString();
|
||||||
|
return String(v);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Stat card ───────────────────────────────────
|
||||||
|
|
||||||
|
function Stat({ label, value }) {
|
||||||
|
return html`
|
||||||
|
<div style="padding:8px 12px;background:var(--bg-secondary, #f5f5f5);border-radius:6px;min-width:120px;">
|
||||||
|
<div style="font-size:11px;color:var(--text-muted, #888);margin-bottom:2px;">${label}</div>
|
||||||
|
<div style="font-size:16px;font-weight:600;font-variant-numeric:tabular-nums;">${value}</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Panels ──────────────────────────────────────
|
||||||
|
|
||||||
|
function PanelHeader({ title }) {
|
||||||
|
return html`<h3 style="margin:0 0 12px;font-size:14px;font-weight:600;text-transform:uppercase;letter-spacing:0.5px;color:var(--text-muted, #888);">${title}</h3>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function RuntimePanel({ m }) {
|
||||||
|
return html`
|
||||||
|
<div style="margin-bottom:24px;">
|
||||||
|
<${PanelHeader} title="Runtime" />
|
||||||
|
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:8px;">
|
||||||
|
<${Stat} label="Uptime" value=${formatDuration(m.uptime_sec)} />
|
||||||
|
<${Stat} label="Goroutines" value=${formatNum(m.goroutines)} />
|
||||||
|
<${Stat} label="Heap Alloc" value=${formatBytes(m.heap_alloc)} />
|
||||||
|
<${Stat} label="Heap Sys" value=${formatBytes(m.heap_sys)} />
|
||||||
|
<${Stat} label="Stack In Use" value=${formatBytes(m.stack_in_use)} />
|
||||||
|
<${Stat} label="GC Cycles" value=${formatNum(m.gc_cycles)} />
|
||||||
|
<${Stat} label="Last GC Pause" value=${formatMs((m.gc_pause_ns || 0) / 1e6)} />
|
||||||
|
<${Stat} label="GC CPU" value=${formatPct(m.gc_cpu_pct)} />
|
||||||
|
<${Stat} label="WS Clients" value=${formatNum(m.ws_clients)} />
|
||||||
|
<${Stat} label="Extensions" value=${formatNum(m.extensions_loaded)} />
|
||||||
|
<${Stat} label="Open FDs" value=${m.open_fds >= 0 ? formatNum(m.open_fds) : '\u2014'} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function DBPanel({ m }) {
|
||||||
|
return html`
|
||||||
|
<div style="margin-bottom:24px;">
|
||||||
|
<${PanelHeader} title="Database" />
|
||||||
|
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:8px;">
|
||||||
|
<${Stat} label="Latency" value=${formatMs(m.latency_ms)} />
|
||||||
|
<${Stat} label="Pool Active" value=${formatNum(m.pool_active)} />
|
||||||
|
<${Stat} label="Pool Idle" value=${formatNum(m.pool_idle)} />
|
||||||
|
<${Stat} label="Pool Max" value=${formatNum(m.pool_max)} />
|
||||||
|
<${Stat} label="Wait Count" value=${formatNum(m.wait_count)} />
|
||||||
|
<${Stat} label="Wait Duration" value=${formatMs(m.wait_duration_ms)} />
|
||||||
|
${m.dead_tuples != null && html`<${Stat} label="Dead Tuples" value=${formatNum(m.dead_tuples)} />`}
|
||||||
|
${m.active_backends != null && html`<${Stat} label="Active Backends" value=${formatNum(m.active_backends)} />`}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function ClusterPanel({ cluster }) {
|
||||||
|
return html`
|
||||||
|
<div style="margin-bottom:24px;">
|
||||||
|
<${PanelHeader} title=${'Cluster \u2014 ' + cluster.size + ' node' + (cluster.size !== 1 ? 's' : '')} />
|
||||||
|
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:12px;">
|
||||||
|
${cluster.nodes.map(n => html`<${NodeCard} key=${n.node_id} node=${n} />`)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function NodeCard({ node }) {
|
||||||
|
const stats = node.stats || {};
|
||||||
|
const ageLabel = node.heartbeat_age_ms < 15000
|
||||||
|
? Math.floor(node.heartbeat_age_ms / 1000) + 's ago'
|
||||||
|
: Math.floor(node.heartbeat_age_ms / 60000) + 'm ago';
|
||||||
|
const ageColor = node.heartbeat_age_ms < 15000 ? 'var(--accent, #2a6)' : 'var(--warning, #c80)';
|
||||||
|
|
||||||
|
return html`
|
||||||
|
<div style="border:1px solid var(--border, #ddd);border-radius:8px;padding:12px;background:var(--bg-primary, #fff);">
|
||||||
|
<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:8px;">
|
||||||
|
<strong style="font-size:13px;">${node.node_id}</strong>
|
||||||
|
<span style="font-size:11px;color:${ageColor};font-weight:600;">${ageLabel}</span>
|
||||||
|
</div>
|
||||||
|
${node.endpoint && html`<div style="font-size:11px;color:var(--text-muted, #888);margin-bottom:8px;">${node.endpoint}</div>`}
|
||||||
|
<div style="display:grid;grid-template-columns:repeat(3,1fr);gap:6px;">
|
||||||
|
<${Stat} label="Uptime" value=${formatDuration(stats.uptime_sec)} />
|
||||||
|
<${Stat} label="WS Clients" value=${formatNum(stats.ws_clients)} />
|
||||||
|
<${Stat} label="Goroutines" value=${formatNum(stats.goroutines)} />
|
||||||
|
<${Stat} label="Heap" value=${formatBytes(stats.heap_alloc)} />
|
||||||
|
<${Stat} label="GC Pause" value=${formatMs((stats.gc_pause_ns || 0) / 1e6)} />
|
||||||
|
<${Stat} label="GC Cycles" value=${formatNum(stats.gc_cycles)} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function ExtensionPanel({ m }) {
|
||||||
|
return html`
|
||||||
|
<div style="margin-bottom:24px;">
|
||||||
|
<${PanelHeader} title="Extensions" />
|
||||||
|
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:8px;">
|
||||||
|
<${Stat} label="Starlark Execs" value=${formatNum(m.starlark_exec_total)} />
|
||||||
|
<${Stat} label="Starlark Errors" value=${formatNum(m.starlark_errors_total)} />
|
||||||
|
<${Stat} label="Avg Duration" value=${formatMs(m.starlark_avg_duration_ms)} />
|
||||||
|
<${Stat} label="Trigger Fires" value=${formatNum(m.trigger_fires_total)} />
|
||||||
|
<${Stat} label="Events Published" value=${formatNum(m.event_bus_published)} />
|
||||||
|
<${Stat} label="Events Delivered" value=${formatNum(m.event_bus_delivered)} />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main component ──────────────────────────────
|
||||||
|
|
||||||
|
export default function HealthSection() {
|
||||||
|
const [data, setData] = useState(null);
|
||||||
|
const [error, setError] = useState(null);
|
||||||
|
const [interval, setInterval_] = useState(10);
|
||||||
|
const timerRef = useRef(null);
|
||||||
|
|
||||||
|
const load = useCallback(async () => {
|
||||||
|
try {
|
||||||
|
const resp = await sw.api.admin.metrics();
|
||||||
|
setData(resp);
|
||||||
|
setError(null);
|
||||||
|
} catch (e) {
|
||||||
|
setError(e.message);
|
||||||
|
}
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
load();
|
||||||
|
timerRef.current = setInterval(load, interval * 1000);
|
||||||
|
return () => clearInterval(timerRef.current);
|
||||||
|
}, [load, interval]);
|
||||||
|
|
||||||
|
if (error && !data) {
|
||||||
|
return html`<div class="settings-placeholder" style="color:var(--error,red);">Failed to load metrics: ${error}</div>`;
|
||||||
|
}
|
||||||
|
if (!data) {
|
||||||
|
return html`<div class="settings-placeholder">Loading metrics\u2026</div>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
return html`
|
||||||
|
<div>
|
||||||
|
<div style="display:flex;justify-content:flex-end;align-items:center;margin-bottom:16px;gap:8px;">
|
||||||
|
<span style="font-size:11px;color:var(--text-muted, #888);">Refresh</span>
|
||||||
|
<select style="font-size:12px;padding:2px 6px;" value=${interval} onChange=${e => setInterval_(+e.target.value)}>
|
||||||
|
<option value="5">5s</option>
|
||||||
|
<option value="10">10s</option>
|
||||||
|
<option value="30">30s</option>
|
||||||
|
<option value="60">60s</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<${RuntimePanel} m=${data.runtime} />
|
||||||
|
<${DBPanel} m=${data.db} />
|
||||||
|
${data.cluster && html`<${ClusterPanel} cluster=${data.cluster} />`}
|
||||||
|
<${ExtensionPanel} m=${data.extensions} />
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
@@ -22,7 +22,7 @@ const ADMIN_SECTIONS = {
|
|||||||
people: ['users', 'teams', 'groups'],
|
people: ['users', 'teams', 'groups'],
|
||||||
workflows: ['workflows'],
|
workflows: ['workflows'],
|
||||||
system: ['settings', 'storage', 'packages', 'connections', 'broadcast', 'backup'],
|
system: ['settings', 'storage', 'packages', 'connections', 'broadcast', 'backup'],
|
||||||
monitoring: ['audit'],
|
monitoring: ['health', 'audit'],
|
||||||
};
|
};
|
||||||
|
|
||||||
const ADMIN_LABELS = {
|
const ADMIN_LABELS = {
|
||||||
@@ -31,7 +31,7 @@ const ADMIN_LABELS = {
|
|||||||
settings: 'Settings', storage: 'Storage', packages: 'Packages',
|
settings: 'Settings', storage: 'Storage', packages: 'Packages',
|
||||||
connections: 'Connections', broadcast: 'Broadcast',
|
connections: 'Connections', broadcast: 'Broadcast',
|
||||||
backup: 'Backup',
|
backup: 'Backup',
|
||||||
audit: 'Audit',
|
health: 'Health', audit: 'Audit',
|
||||||
};
|
};
|
||||||
|
|
||||||
// ── Extension config sections ──────
|
// ── Extension config sections ──────
|
||||||
@@ -52,7 +52,7 @@ const CATEGORY_META = {
|
|||||||
people: { label: 'People', first: 'users', icon: 'M17 21v-2a4 4 0 0 0-4-4H5a4 4 0 0 0-4 4v2|C9 7 4' },
|
people: { label: 'People', first: 'users', icon: 'M17 21v-2a4 4 0 0 0-4-4H5a4 4 0 0 0-4 4v2|C9 7 4' },
|
||||||
workflows: { label: 'Workflows', first: 'workflows', icon: 'P16 3 21 3 21 8|L4 20 21 3|P21 16 21 21 16 21|L15 15 21 21|L4 4 9 9' },
|
workflows: { label: 'Workflows', first: 'workflows', icon: 'P16 3 21 3 21 8|L4 20 21 3|P21 16 21 21 16 21|L15 15 21 21|L4 4 9 9' },
|
||||||
system: { label: 'System', first: 'settings', icon: 'C12 12 3|M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-4 0v-.09' },
|
system: { label: 'System', first: 'settings', icon: 'C12 12 3|M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-4 0v-.09' },
|
||||||
monitoring: { label: 'Monitoring', first: 'audit', icon: 'L18 20 18 10|L12 20 12 4|L6 20 6 14' },
|
monitoring: { label: 'Monitoring', first: 'health', icon: 'L18 20 18 10|L12 20 12 4|L6 20 6 14' },
|
||||||
};
|
};
|
||||||
|
|
||||||
// ── Lazy section imports ────────────────────
|
// ── Lazy section imports ────────────────────
|
||||||
@@ -69,6 +69,7 @@ const sectionModules = {
|
|||||||
connections: () => import(`./connections.js${_v}`),
|
connections: () => import(`./connections.js${_v}`),
|
||||||
broadcast: () => import(`./broadcast.js${_v}`),
|
broadcast: () => import(`./broadcast.js${_v}`),
|
||||||
backup: () => import(`./backup.js${_v}`),
|
backup: () => import(`./backup.js${_v}`),
|
||||||
|
health: () => import(`./health.js${_v}`),
|
||||||
audit: () => import(`./audit.js${_v}`),
|
audit: () => import(`./audit.js${_v}`),
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user