core/docs/AUDIT-0.22.0-PREFLIGHT.md

Pre-Flight Audit — v0.22.0 Kickoff

Date: 2026-03-01 Current version: 0.21.6 Test instance: switchboard-be-test in gobha-ai-chat namespace Schema: 012_v0214_git.sql — up to date

---

1. Bug: Channel List Corruption (SafeJSON 500)

Symptoms

23:24:44 ⚠ SafeJSON: marshal failed: json: error calling MarshalJSON for type
  json.RawMessage: invalid character 'i' looking for beginning of value
23:24:44 | 500 | GET "/test/api/v1/channels?page=1&per_page=100&type=direct"

One channel in the test database has invalid JSON in its settings column. The content starts with the character i (likely a bare word like invalid, inactive, or in_progress — not valid JSON). This kills the entire paginated channel list with a 500.

Root Cause Analysis

Defense layer 1 — scanJSON: The ListChannels scan path at channels.go:236 correctly uses scanJSON(&ch.Settings), which calls json.Valid() and defaults to {} on failure. This should catch corrupt data.

Defense layer 2 — SafeJSON: Pre-marshals the entire response and returns a clean 500 instead of truncated output. This layer IS firing, meaning layer 1 did NOT catch the problem.

Inconsistent scan paths: The Postgres CreateChannel RETURNING path at channels.go:323 scans &ch.Settings directly WITHOUT scanJSON:

// line 323 — Postgres CreateChannel
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,

Compare to the SQLite path at line 306 which correctly uses scanJSON.

How the data got corrupted: Unknown without DB access. Postgres JSONB columns reject invalid JSON at write time (::jsonb cast), so the corruption likely predates the current schema — possibly from a migration, manual DB edit, or an older code version that used TEXT instead of JSONB. Run this to find it:

SELECT id, title, settings
FROM channels
WHERE user_id = '664c71e8-cd8b-47fb-b421-9c48abb815ba'
  AND settings::text NOT LIKE '{%'
  AND settings::text NOT LIKE '[%'
  AND settings::text NOT LIKE '"null"';

Fixes Required

P0 — Fix the data (immediate):

UPDATE channels
SET settings = '{}'::jsonb
WHERE settings IS NULL
   OR NOT (settings::text ~ '^[\[{"]');

P0 — Consistent scanJSON usage (channels.go:323):

// Before (Postgres CreateChannel):
pq.Array(&tags), &ch.Settings, &ch.CreatedAt, &ch.UpdatedAt,

// After:
pq.Array(&tags), scanJSON(&ch.Settings), &ch.CreatedAt, &ch.UpdatedAt,

P1 — Graceful list degradation: When SafeJSON encounters a corrupt item in a paginated list, the current behavior drops the entire response. Instead, add a SafeMarshalList helper that:

1. Marshals items individually
2. Skips corrupt items with a warning log (include the row ID)
3. Returns the healthy items with a warnings field in the response

P1 — Input validation on settings write (channels.go:487):

if req.Settings != nil {
    if !json.Valid([]byte(*req.Settings)) {
        c.JSON(http.StatusBadRequest, gin.H{"error": "settings must be valid JSON"})
        return
    }
    // ... existing merge logic
}

P2 — Audit all RawMessage scan sites: Search for &ch.Settings, &*.Settings, or any json.RawMessage scan that doesn't go through scanJSON. Every JSONB column scanned into json.RawMessage must use the wrapper.

Additional Log Noise

The git endpoints are returning 400s for workspaces without git repos:

23:23:40 | 400 | GET ".../git/status"
23:23:56 | 400 | GET ".../git/branches"

These are expected (workspace has no .git directory), but the frontend should suppress these calls when the workspace has no git_remote_url configured. Frontend fix: check workspace.git_remote_url before calling git endpoints.

---

2. CI Variables → Deployment Audit

Cross-referencing CI secrets/variables (from Gitea) against backend.yaml, config.go, and runtime logs.

Secrets (Gitea Secrets → K8s Secrets)

CI Secret	K8s Secret	Key	Status
ENCRYPTION_KEY	switchboard-encryption	ENCRYPTION_KEY	✅ Active
POSTGRES_USER	switchboard-db-credentials	POSTGRES_USER	✅ Active
POSTGRES_PASSWORD	switchboard-db-credentials	POSTGRES_PASSWORD	✅ Active
POSTGRES_ADMIN_USER	—	—	⚠ Unused by app — likely for CI DB bootstrap
POSTGRES_ADMIN_PASSWORD	—	—	⚠ Unused by app — likely for CI DB bootstrap
S3_ACCESS_KEY	switchboard-s3	S3_ACCESS_KEY	✅ Active
S3_SECRET_KEY	switchboard-s3	S3_SECRET_KEY	✅ Active
SWITCHBOARD_ADMIN_EMAIL	switchboard-admin	email	✅ Active
SWITCHBOARD_ADMIN_PASSWORD	switchboard-admin	password	✅ Active
SWITCHBOARD_ADMIN_USERNAME	switchboard-admin	username	✅ Active

Variables (Gitea Variables → envsubst / K8s env)

CI Variable	Used In	Value	Status
DOMAIN	ingress.yaml	gobha.ai	✅
NAMESPACE	all manifests	gobha-ai-chat	✅
POSTGRES_HOST	backend.yaml env	postgres-primary.postgres.svc.cluster.local	✅
PROVIDER	?	venice	⚠ Orphan — not referenced in any manifest or config.go
S3_BUCKET	switchboard-s3 secret	integration-test	✅
S3_ENDPOINT	switchboard-s3 secret	http://192.168.10.29:7480	✅
SEED_PROVIDERS	backend.yaml env (direct)	venice:...:micro	✅ New (Mar 1)
SEED_USERS	switchboard-seed-users secret	darthvader:...,lukeskywalker:...	✅
STORAGE_BACKEND	backend.yaml env	s3	✅
STORAGE_CLASS	storage-pvc.yaml	cephfs	✅
STORAGE_SIZE	storage-pvc.yaml	(visible, value cut off)	✅

Variables Expected by backend.yaml but NOT in CI Screenshots

These are likely set in the CI pipeline YAML directly (not as repo variables):

Variable	Used In	Expected Source
ENVIRONMENT	backend.yaml env	Pipeline per-branch
GIN_MODE	backend.yaml env	Pipeline (release/debug)
BASE_PATH	backend.yaml env	Pipeline per-deploy (/test, /prod, etc.)
POSTGRES_PORT	backend.yaml env	Default 5432 (config.go fallback)
DB_NAME	backend.yaml env	Pipeline per-deploy
DEPLOY_SUFFIX	all manifests	Pipeline (-test, -prod, etc.)
BE_IMAGE	backend.yaml image	Pipeline / registry
IMAGE_TAG	backend.yaml image	Pipeline / git SHA
BE_REPLICAS	backend.yaml replicas	Pipeline
BE_MEMORY_REQUEST	backend.yaml resources	Pipeline
BE_MEMORY_LIMIT	backend.yaml resources	Pipeline
BE_CPU_REQUEST	backend.yaml resources	Pipeline
BE_CPU_LIMIT	backend.yaml resources	Pipeline
EXTRACTION_CONCURRENCY	backend.yaml env	Pipeline (default 3)

Issues Found

1. PROVIDER variable is orphaned. Value is "venice" but nothing in backend.yaml or config.go references it. Likely a legacy variable from before SEED_PROVIDERS was added. Clean up or document.

2. SEED_PROVIDERS contains an API key in a CI Variable (not Secret). The screenshot shows the Venice API key visible in plaintext in the Variables section. This should be moved to Gitea Secrets for proper masking. The backend.yaml needs a corresponding mapping (currently SEED_PROVIDERS may be injected via envsubst or direct env, verify the CI pipeline).

3. SEED_USERS passwords are plaintext in CI. password1234 for both test users. Acceptable for test environment but should be documented as test-only. Production pipeline should NOT have SEED_USERS set at all (and SeedProviders already blocks production).

4. Missing SEED_PROVIDERS in backend.yaml env block. The K8s manifest doesn't have a SEED_PROVIDERS env var entry. It works because the CI pipeline likely injects it via envsubst into a direct env entry, but this should be made explicit in backend.yaml for documentation parity.

5. No JWT_SECRET in CI secrets. Config.go defaults to "dev-secret-change-me". The test instance may be running with this default. Add a proper secret for test environments.

6. No S3_REGION or S3_PREFIX in CI. Config.go defaults to us-east-1 and "" respectively. Fine for Ceph RGW but should be documented.

Recommendations

• Move SEED_PROVIDERS from CI Variable to CI Secret
• Remove or document the orphaned PROVIDER variable
• Add JWT_SECRET to CI Secrets and wire through backend.yaml
• Add SEED_PROVIDERS env entry to backend.yaml (with secretKeyRef)
• Add a CI pipeline comment documenting which variables are set inline vs. in repo settings

---

3. Test Hardening — Unhappy Paths

Current test coverage is strong on the happy path (164+ integration tests). The corruption bug exposes gaps in unhappy-path coverage.

Missing Test Categories

A. Data corruption resilience (integration_test.go):

• Corrupt JSONB in channels: INSERT a channel with valid settings, then UPDATE settings directly via SQL with invalid text. Verify GET /channels still returns 200 with the corrupt channel included (settings defaulted to {}).
• Corrupt JSONB in projects: Same pattern for project settings.
• Corrupt JSONB in teams: Same for team settings/policies.
• NULL settings column: Verify channels with NULL settings serialize correctly (should be {} or null, not crash).
• Corrupt tags column: Insert channel with malformed tags array. Verify list doesn't 500.

B. Settings write validation:

• Invalid JSON in channel settings update: PUT /channels/:id with {"settings": "not-json"} — should 400, not write to DB.
• Oversized settings: PUT with a 10MB settings blob — should be rejected.
• SQL injection via settings merge: PUT with settings containing SQL metacharacters — verify parameterized query prevents injection.

C. Scan alignment tests (safe_json_test.go):

• Column count mismatch: Mock a query result with wrong column count, verify clean error instead of panic.
• Type mismatch: Scan a TEXT column into json.RawMessage without scanJSON — verify SafeJSON catches it.
• Driver buffer reuse: Scan multiple rows with RawMessage, verify each row's data is independent (copy, not reference).

D. Startup resilience (config/seed tests):

• SEED_PROVIDERS with invalid format: "openai" (no key), ":", ":::", empty string — verify graceful skip.
• SEED_PROVIDERS with unknown provider: "unknown:key123" — verify warning log, not crash.
• SEED_USERS with invalid format: Malformed CSV — verify skip.
• DATABASE_URL with unreachable host: Verify startup logs error and serves 503 on API endpoints.

E. WebSocket reconnection:

• Rapid disconnect/reconnect: The logs show multiple ws connect/ disconnect cycles in quick succession. Verify no goroutine leaks, no duplicate event delivery.
• Auth token expiry during active connection: Verify clean disconnect with appropriate error, not silent data loss.

F. Frontend error handling (manual or e2e):

• 500 on channel list: Frontend should show error state, not blank screen. Currently unknown behavior.
• 400 on git endpoints for non-git workspace: Frontend should not call git endpoints when workspace has no git_remote_url.
• Network timeout during streaming completion: Verify partial response is preserved, user can retry.

Test Infrastructure Needed

1. SQL fixture injection: Helper function to INSERT corrupt data directly via SQL, bypassing store layer validation. For corruption resilience tests.

2. SafeJSON audit script: grep -rn '&.*\.Settings\|json.RawMessage' to find all RawMessage scan sites. Run as CI lint check to catch regressions.

3. Integration test for each migration: Both fresh-install path AND upgrade path from previous version. Currently tested manually — automate.

---

4. Quick Wins Before v0.22.0 Work Starts

Priority-ordered items that should land as v0.21.7 patches:

1. Fix corrupt data in test DB — SQL UPDATE (5 min)
2. Add scanJSON to CreateChannel Postgres path — one-line fix (5 min)
3. Add json.Valid check on settings write — channels.go UpdateChannel (10 min)
4. Move SEED_PROVIDERS to CI Secret — CI UI change (5 min)
5. Suppress git endpoint calls for non-git workspaces — frontend (15 min)
6. Add corruption resilience integration tests — 2-3 tests (1 hr)
7. Add JWT_SECRET to CI Secrets — CI + backend.yaml (15 min)