switchboard/core
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Releases 9 Wiki Activity

docs: ROADMAP full restore + #98-143 tech debt links + archive historical DESIGN/CHANGES (#144)

Browse Source
This commit is contained in:
xcaliber
2026-03-02 13:14:20 +00:00
parent 14c691b52f
commit 9940fb5831
20 changed files with 2422 additions and 4570 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1304
docs/DESIGN-0.12.0.md
View File

File diff suppressed because it is too large Load Diff

231
docs/DESIGN-0.13.1.md
View File

@@ -1,231 +0,0 @@
# DESIGN-0.13.1 — Web Search + URL Fetch + Tool Toggle
## Overview
Built-in `web_search` and `url_fetch` tools using the existing tool framework (v0.11.0),
plus a chat-bar tools toggle menu so users can enable/disable tool categories per-session.
Depends on: tool framework (v0.11.0), admin panel (v0.13.0).
---
## 1. Tool Categories
Add a `Category` field to `ToolDef` so tools self-declare their group.
The frontend uses categories for the toggle menu; the backend ignores them.
| Category | Tools | Default |
|-----------|---------------------------------------------|---------|
| search | `web_search`, `url_fetch` | ON |
| notes | `note_create`, `note_search`, `note_update`, `note_list` | ON |
| utilities | `datetime`, `calculator` | ON |
| browser | (extension-defined: `js_eval`, `regex_test`)| ON |
```go
type ToolDef struct {
Name string `json:"name"`
DisplayName string `json:"display_name,omitempty"` // human-readable label for UI
Description string `json:"description"`
Parameters json.RawMessage `json:"parameters"`
Category string `json:"category,omitempty"` // NEW
}
```
## 2. Per-Request Tool Filtering
### Request field
```json
{ "disabled_tools": ["web_search", "url_fetch"] }
```
### Backend filter
`buildToolDefs()` skips tools whose name appears in `disabled_tools`.
Zero API changes — the field is optional, empty = all tools enabled.
### Frontend state
`localStorage` key: `cs-disabled-tools` → JSON array of tool names.
The tools toggle menu flips categories on/off which maps to individual tool names.
## 3. web_search Tool
### Search Provider Interface
```go
type SearchProvider interface {
Search(ctx context.Context, query string, maxResults int) ([]SearchResult, error)
Name() string
}
type SearchResult struct {
Title string `json:"title"`
URL string `json:"url"`
Snippet string `json:"snippet"`
}
```
### DuckDuckGo Provider (default — no API key)
Uses the DuckDuckGo HTML search endpoint. No rate-limit key required.
Falls back gracefully if blocked (returns error result to LLM, doesn't crash).
### SearXNG Provider (self-hosted)
Configurable endpoint + optional API key. JSON API (`/search?format=json`).
Good fit for airgapped deployments.
### Admin Config
Stored in `global_config` table (existing):
- `search_provider`: `"duckduckgo"` | `"searxng"` (default: `"duckduckgo"`)
- `search_endpoint`: SearXNG URL (only used when provider=searxng)
- `search_max_results`: `5` (default)
Admin UI: System → Settings section (new "Search" subsection).
### Tool Schema
```json
{
"name": "web_search",
"description": "Search the web for current information. Returns titles, URLs, and snippets.",
"category": "search",
"parameters": {
"type": "object",
"properties": {
"query": { "type": "string", "description": "Search query" },
"max_results": { "type": "integer", "description": "Max results (1-10, default 5)" }
},
"required": ["query"]
}
}
```
## 4. url_fetch Tool
HTTP GET with content extraction. Reuses the text extraction pattern
from file handling (v0.12.0) where possible.
### Behavior
1. HTTP GET with reasonable timeout (15s) and User-Agent
2. Follow redirects (max 3)
3. Extract readable text content (strip HTML tags, scripts, styles)
4. Truncate to ~8000 chars to avoid flooding the context
5. Return title + extracted text + metadata
### Safety
- Respect robots.txt? No — the LLM is fetching on behalf of the user, not crawling.
- Block private/internal IPs (10.x, 192.168.x, 127.x, ::1) to prevent SSRF.
- Configurable domain allowlist/blocklist in admin settings (future).
### Tool Schema
```json
{
"name": "url_fetch",
"description": "Fetch and extract readable text content from a URL.",
"category": "search",
"parameters": {
"type": "object",
"properties": {
"url": { "type": "string", "description": "URL to fetch" },
"max_length": { "type": "integer", "description": "Max content length in chars (default 8000)" }
},
"required": ["url"]
}
}
```
## 5. Frontend: Tools Toggle Menu
### UI Position
Chat input bar, left side, next to the attach button. Wrench/tool icon.
```
┌──────────────────────────────────────┐
│ 📎 🔧 [message input...] [Send]│
└──────────────────────────────────────┘
▲
┌────────────────┐
│ ☑ Web Search │
│ ☑ Notes │
│ ☑ Utilities │
│ ☑ Extensions │
└────────────────┘
```
### Behavior
- Click tool icon → popup above (using existing popup-menu pattern)
- Each category row has a toggle checkbox + expand chevron
- **Category toggle**: master on/off for all tools in the category
- **Expand chevron**: shows individual tool rows indented below
- **Tri-state checkbox**: if some tools in a category are disabled, category shows indeterminate (─)
- Toggling a category adds/removes all its tool names from `cs-disabled-tools`
- Toggling an individual tool adds/removes just that name
- State persists across page reloads via localStorage
- Disabled tools sent as `disabled_tools` array in completion request
- Tool icon shows visual indicator when any category is disabled
- If model doesn't support tool_calling, icon is hidden/grayed
```
┌────────────────────────┐
│ ☑ 🔍 Web Search › │ ← click › to expand
│ ☑ Search │
│ ☑ Fetch URL │
│ ☑ 📝 Notes › │
│ ☐ 🧮 Utilities › │ ← category off = all children off
│ ☑ 🧩 Extensions › │
└────────────────────────┘
```
### Tool Manifest
Frontend needs to know the category mapping and display names.
**API endpoint** — `GET /api/v1/tools` returns `[{name, display_name, category, description}]`
The frontend caches this on login. `display_name` provides human-readable labels
(e.g. "Search" instead of "web_search", "Create" instead of "note_create").
### New API Endpoint
```
GET /api/v1/tools → [{name, description, category}]
```
Returns server-side tools + browser extension tool schemas.
Used by frontend to build the toggle menu dynamically.
## 6. File Structure
### New backend files
- `server/tools/websearch.go` — web_search tool + search provider interface
- `server/tools/urlfetch.go` — url_fetch tool
- `server/tools/search/duckduckgo.go` — DuckDuckGo provider
- `server/tools/search/searxng.go` — SearXNG provider
- `server/tools/search/provider.go` — interface + registry
### Modified backend files
- `server/tools/types.go` — add Category to ToolDef
- `server/tools/registry.go` — add AllDefinitionsFiltered(disabled)
- `server/handlers/completion.go` — accept disabled_tools, filter in buildToolDefs
- `server/handlers/messages.go` — same filtering
- `server/handlers/routes.go` — add GET /api/v1/tools endpoint
- `server/tools/datetime.go` — add Category: "utilities"
- `server/tools/calculator.go` — add Category: "utilities"
- `server/tools/notes.go` — add Category: "notes"
### New/modified frontend files
- `src/index.html` — tools toggle button in input-wrap
- `src/css/styles.css` — tools popup styling
- `src/js/chat.js` — send disabled_tools in completion, wire toggle
- `src/js/api.js` — add disabled_tools param, add getTools() method
## 7. Phased Delivery
**Phase 1: Backend tools + filtering**
- ToolDef.Category, disabled_tools filtering
- web_search + url_fetch tools
- DuckDuckGo provider
- GET /api/v1/tools endpoint
- Tests
**Phase 2: Frontend toggle**
- Tools toggle popup menu
- localStorage persistence
- Completion request wiring
- CSS
**Phase 3: Admin config + SearXNG**
- Search provider settings in admin panel
- SearXNG provider implementation
- Domain filtering

845
docs/DESIGN-0.14.0.md
View File

@@ -1,845 +0,0 @@
# DESIGN-0.14.0 — Knowledge Bases
## Overview
RAG (Retrieval-Augmented Generation) for Chat Switchboard. Users upload
documents into named knowledge bases, the backend chunks and embeds them
via the embedding model role (v0.10.0), stores vectors in pgvector, and a
`kb_search` tool lets the LLM pull relevant context at completion time.
Depends on: embedding model role (v0.10.0), file storage (v0.12.0),
tool framework (v0.11.0), admin panel (v0.13.0).
---
## 0. Pre-Req Fix: Embedding Model Selection
The embedding role dropdown is broken — `filterModels()` in
`ui-primitives.js` filters by `model_type === 'embedding'`, but many
providers don't report a type for embedding models (defaults to `'chat'`
and gets filtered out). This blocks KB setup entirely.
**Fix (ship before or with Phase 1):**
1. **Manual model ID entry** — add a text input fallback alongside the
dropdown. If the dropdown shows no results for the selected provider,
display an editable text field pre-populated with the `model_id` from
the saved binding (if any). Users type the model ID directly
(e.g. `text-embedding-3-small`). The save handler accepts either
dropdown selection or manual text entry.
2. **Tolerant type filter** — change `filterModels()` to include models
where `model_type` is empty/null/undefined when the role's type filter
is `'embedding'`. Embedding models are rare enough that showing
untyped models alongside typed ones is better than showing nothing.
```javascript
// Before (strict — breaks when model_type missing)
(m.model_type || 'chat') === typeFilter
// After (tolerant for embedding role)
typeFilter === 'embedding'
? (!m.model_type || m.model_type === 'embedding')
: (m.model_type || 'chat') === typeFilter
```
3. **Combo UI** — the slot renderer gets a hybrid dropdown+input:
```
┌──────────────────────────────────────────────┐
│ Provider: [OpenAI ▾] │
│ Model: [text-embedding-3-small ▾] [✎] │
│ ↑ toggle │
│ [text-embedding-3-small ] │ ← manual entry (shown on ✎ click)
└──────────────────────────────────────────────┘
```
The pencil icon toggles between dropdown and text input. If the dropdown
has zero options after provider change, auto-switch to manual entry.
Saved bindings with model IDs not in the dropdown auto-show in manual
mode on load.
---
## 1. Schema — Migration `008_knowledge_bases.sql`
### pgvector Extension
```sql
CREATE EXTENSION IF NOT EXISTS vector;
```
Requires `pgvector` installed on the PostgreSQL server. The migration
checks for the extension and fails with a clear error if unavailable,
rather than silently degrading. The README/deployment docs will note the
dependency.
**Dimension handling:** Embedding dimensions vary by model
(OpenAI ada-002 = 1536, text-embedding-3-small = 1536,
text-embedding-3-large = 3072, many open-source = 768 or 1024).
The dimension is stored per KB and the vector column uses the max
supported (3072), with vectors zero-padded or truncated at insert time.
Alternative: use `halfvec` for storage efficiency. We go with a single
fixed column at 3072 and document the tradeoff.
> **Decision:** 3072 is generous but future-proof. If storage becomes
> an issue, we add a `dimensions` column to `knowledge_bases` and
> migrate to per-dimension indexes later. For v0.14.0, KISS.
### Tables
```sql
-- ── Knowledge Bases ──────────────────────────
CREATE TABLE knowledge_bases (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
scope TEXT NOT NULL DEFAULT 'global', -- global, team, personal
owner_id UUID, -- user_id (personal) or NULL
team_id UUID REFERENCES teams(id) ON DELETE CASCADE,
embedding_config JSONB NOT NULL DEFAULT '{}', -- snapshot: { provider_config_id, model_id, dimensions }
-- Stats (denormalized, updated on ingest)
document_count INT NOT NULL DEFAULT 0,
chunk_count INT NOT NULL DEFAULT 0,
total_bytes BIGINT NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'active', -- active, processing, error
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT kb_scope_check CHECK (
(scope = 'global' AND owner_id IS NULL) OR
(scope = 'team' AND team_id IS NOT NULL) OR
(scope = 'personal' AND owner_id IS NOT NULL)
)
);
CREATE INDEX idx_kb_scope ON knowledge_bases(scope);
CREATE INDEX idx_kb_owner ON knowledge_bases(owner_id) WHERE owner_id IS NOT NULL;
CREATE INDEX idx_kb_team ON knowledge_bases(team_id) WHERE team_id IS NOT NULL;
-- ── KB Documents ─────────────────────────────
-- Metadata row per uploaded document. Blobs live in ObjectStore
-- under key: kb/{kb_id}/{doc_id}_{filename}
CREATE TABLE kb_documents (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
storage_key TEXT NOT NULL, -- ObjectStore key
extracted_text TEXT, -- full extracted text (for re-chunking)
chunk_count INT NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'pending', -- pending, chunking, embedding, ready, error
error TEXT,
uploaded_by UUID NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_kbdoc_kb ON kb_documents(kb_id);
-- ── KB Chunks ────────────────────────────────
-- Chunked text + embedding vector for similarity search.
CREATE TABLE kb_chunks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
document_id UUID NOT NULL REFERENCES kb_documents(id) ON DELETE CASCADE,
chunk_index INT NOT NULL, -- ordinal within document
content TEXT NOT NULL, -- chunk text
token_count INT NOT NULL DEFAULT 0,
embedding vector(3072), -- pgvector column
metadata JSONB NOT NULL DEFAULT '{}', -- { page, section, heading, ... }
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_kbchunk_kb ON kb_chunks(kb_id);
CREATE INDEX idx_kbchunk_doc ON kb_chunks(document_id);
-- IVFFlat index for similarity search
-- (created after initial data load; needs rows to train)
-- Deferred: created by ingest handler after first batch
-- CREATE INDEX idx_kbchunk_embedding ON kb_chunks
-- USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
-- ── Channel-KB Links ─────────────────────────
-- Which KBs are active for a given channel.
CREATE TABLE channel_knowledge_bases (
channel_id UUID NOT NULL REFERENCES channels(id) ON DELETE CASCADE,
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
enabled BOOLEAN NOT NULL DEFAULT true,
added_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (channel_id, kb_id)
);
```
### Scope Rules
| Scope | Visible to | Created by | Embedding cost |
|----------|------------------------|-------------------|-----------------|
| global | All users | Admin | Org provider |
| team | Team members | Team admin | Team provider |
| personal | Owner only | BYOK user | User's own key |
Personal KBs follow the same BYOK pattern as personal providers — the
user's UEK decrypts their embedding provider key. Embedding costs are
logged to usage_log with `role = 'embedding'`.
---
## 2. Chunking Pipeline
### Strategy: Recursive Character Splitter
Start simple, iterate. Recursive splitting with configurable chunk size
and overlap. No semantic chunking in v0.14.0 (requires embedding every
boundary candidate — expensive, diminishing returns for a first pass).
```go
// server/knowledge/chunker.go
type ChunkConfig struct {
ChunkSize int // target chars per chunk (default 1000)
ChunkOverlap int // overlap chars between chunks (default 200)
Separators []string // split hierarchy: ["\n\n", "\n", ". ", " "]
}
type Chunk struct {
Content string
Index int // ordinal in document
TokenCount int // estimated token count
Metadata map[string]any // page, heading context
}
func SplitText(text string, cfg ChunkConfig) []Chunk
```
**Default config:** 1000 chars, 200 overlap (~250 tokens per chunk).
These defaults are stored in `knowledge_bases.embedding_config` and
overridable per-KB via the admin UI. The chunker tries separators in
order: paragraph breaks first, then line breaks, then sentences, then
words.
### Text Extraction Reuse
Documents uploaded to KBs use the same extraction pipeline as
attachments (v0.12.0). For inline-extractable types (TXT, MD, CSV),
text is read directly. For sidecar types (PDF, DOCX), the extraction
queue processes them. The extracted text is stored in
`kb_documents.extracted_text` for re-chunking if config changes.
---
## 3. Embedding Pipeline
### Flow
```
Upload → Extract Text → Chunk → Embed (batch) → Store vectors
```
### Batch Embedding
The embedding role's `Embed()` method already accepts `[]string` input.
Chunks are batched in groups of 100 (or fewer, respecting provider token
limits). Each batch is a single API call.
```go
// server/knowledge/embedder.go
type Embedder struct {
roleResolver *roles.Resolver
}
// EmbedChunks generates embeddings for a slice of chunks.
// Uses the embedding role with the resolution chain:
// personal → team → global (same as other role consumers).
func (e *Embedder) EmbedChunks(ctx context.Context, userID string,
teamID *string, chunks []Chunk) ([][]float64, error)
```
### Dimension Normalization
The embedding response includes raw vectors. If the vector dimension
is less than 3072 (the column width), zero-pad. If greater, truncate
(unlikely but defensive). Store the actual dimension in
`knowledge_bases.embedding_config.dimensions` so search queries can
account for it.
> **Note:** Mixing embedding models within a KB is invalid — vectors
> from different models aren't comparable. The `embedding_config` is
> snapshotted at KB creation. Changing the embedding model requires
> re-embedding all documents (explicit admin action).
---
## 4. Ingestion Flow
No background job system exists yet (that's v0.15.0 compaction).
Ingestion is **synchronous with progress polling** — same pattern as
the extraction queue (v0.12.0).
### API Endpoints
```
POST /api/v1/knowledge-bases — Create KB
GET /api/v1/knowledge-bases — List (scoped)
GET /api/v1/knowledge-bases/:id — Get KB details
PUT /api/v1/knowledge-bases/:id — Update name/description
DELETE /api/v1/knowledge-bases/:id — Delete KB + all docs/chunks
POST /api/v1/knowledge-bases/:id/documents — Upload document(s)
GET /api/v1/knowledge-bases/:id/documents — List documents
DELETE /api/v1/knowledge-bases/:id/documents/:docId — Delete document + chunks
GET /api/v1/knowledge-bases/:id/documents/:docId/status — Poll ingestion status
POST /api/v1/knowledge-bases/:id/rebuild — Re-chunk + re-embed all docs
POST /api/v1/knowledge-bases/:id/search — Direct search (debug/admin)
```
Admin endpoints mirror the pattern under `/api/v1/admin/knowledge-bases/`
for global KBs. Team KBs under `/api/v1/teams/:teamId/knowledge-bases/`.
Personal KBs under the base path (scoped by auth).
### Upload + Ingest Sequence
```
Client Server
│ │
├── POST /kb/:id/documents ───► │ Save blob to ObjectStore
│ (multipart/form-data) │ Create kb_documents row (status=pending)
│ │ Return { document_id, status: "pending" }
│ │
│ │── goroutine: ingestDocument()
│ │ ├─ Extract text (inline or queue)
│ │ ├─ status → "chunking"
│ │ ├─ Chunk text
│ │ ├─ status → "embedding"
│ │ ├─ Batch embed chunks
│ │ ├─ INSERT kb_chunks with vectors
│ │ ├─ Update KB stats
│ │ └─ status → "ready"
│ │
├── GET /kb/:id/docs/:d/status► │ Return current status + progress
│ (poll every 2s) │ { status, chunk_count, error }
◄───────────────────────────── │
```
The goroutine is fire-and-forget per document. The status column on
`kb_documents` is the progress indicator. Frontend polls until
`status = 'ready'` or `status = 'error'`.
**Concurrency:** A per-process semaphore (channel of size 3) limits
concurrent ingestion goroutines. Additional uploads queue behind the
semaphore. This prevents overwhelming the embedding provider with
parallel batch requests.
---
## 5. Search — `kb_search` Tool
### Tool Definition
```go
// server/tools/kbsearch.go
var kbSearchDef = ToolDef{
Name: "kb_search",
DisplayName: "Knowledge Base",
Description: "Search knowledge bases for relevant information. " +
"Returns text passages from uploaded documents that match the query.",
Category: "knowledge",
Parameters: JSONSchema(map[string]interface{}{
"query": Prop("string", "Search query — use natural language"),
"max_results": map[string]interface{}{
"type": "integer",
"description": "Maximum results to return (1-20, default 5)",
},
}, []string{"query"}),
}
```
### Execution
```go
func (t *KBSearchTool) Execute(ctx context.Context,
execCtx ExecutionContext, argsJSON string) (string, error) {
// 1. Parse args
// 2. Get active KBs for this channel
// (from channel_knowledge_bases where enabled = true)
// PLUS personal KBs owned by user (always available)
// 3. Embed the query using the embedding role
// 4. Similarity search across all active KB chunks
// 5. Format results with source attribution
}
```
### Similarity Query
```sql
SELECT c.content, c.metadata, d.filename, kb.name as kb_name,
1 - (c.embedding <=> $1::vector) AS similarity
FROM kb_chunks c
JOIN kb_documents d ON c.document_id = d.id
JOIN knowledge_bases kb ON c.kb_id = kb.id
WHERE c.kb_id = ANY($2) -- active KB IDs
AND 1 - (c.embedding <=> $1) > $3 -- similarity threshold (0.3 default)
ORDER BY c.embedding <=> $1
LIMIT $4; -- max_results
```
### Result Format
```json
{
"results": [
{
"content": "Chunk text here...",
"source": "quarterly-report.pdf",
"kb": "Q4 Reports",
"similarity": 0.87,
"metadata": { "page": 12 }
}
],
"query": "revenue growth Q4",
"searched_kbs": ["Q4 Reports", "Financial Data"]
}
```
### ExecutionContext Extension
The `kb_search` tool needs access to the store layer and role resolver,
which the current `ExecutionContext` doesn't provide. Two options:
**Option A: Expand ExecutionContext** — add optional fields:
```go
type ExecutionContext struct {
UserID string
ChannelID string
// v0.14.0 additions
Stores *store.Stores // nil for tools that don't need it
RoleResolver *roles.Resolver // nil for tools that don't need it
TeamID *string // user's team (for role resolution)
}
```
**Option B: Closure injection** — the tool struct captures dependencies
at registration time:
```go
type kbSearchTool struct {
stores store.Stores
roleResolver *roles.Resolver
}
func init() {
// Deferred registration — called from main.go after stores init
}
```
**Decision: Option B (closure injection).** It's the same pattern the
notes tool already uses (it captures the NoteStore). The `kb_search`
tool is registered after stores and role resolver init in `main.go`,
not via `init()`. Add a `RegisterLate()` function or explicit call.
---
## 6. Context Injection
When `kb_search` returns results, they flow through the existing tool
call loop in `stream_loop.go` — no special injection needed. The tool
result becomes a `role: "tool"` message in the conversation, and the
LLM sees the retrieved context naturally.
### Automatic vs. Tool-Based
Two approaches to getting KB context into the conversation:
| Approach | Pros | Cons |
|----------|------|------|
| **Tool-based** (v0.14.0) | LLM decides when to search; transparent; works with existing tool loop | Extra round-trip; LLM might not search when it should |
| **Auto-inject** (future) | Always available; no tool call overhead | Wastes tokens if not needed; opaque to user |
**v0.14.0: Tool-based only.** The LLM calls `kb_search` when it needs
information. Auto-injection (pre-pend top-K results to every prompt) is
a v0.15.x consideration when we have compaction to manage context
budget.
### System Prompt Hint
When KBs are active on a channel, append a hint to the system prompt:
```
You have access to the following knowledge bases:
- "Q4 Reports" (3 documents)
- "Engineering Wiki" (12 documents)
Use the kb_search tool to find relevant information from these sources.
```
This nudges the model to use the tool without auto-injecting content.
---
## 7. Notes Embedding
Notes already have full-text search via PostgreSQL `tsvector`. Adding
vector search means notes can be found semantically, not just by keyword.
**Approach:** Treat notes as single-chunk documents. On note
create/update, embed the full note text and store in a new column:
```sql
ALTER TABLE notes ADD COLUMN embedding vector(3072);
```
The `note_search` tool (already exists) gains an optional
`semantic: true` parameter that switches from `tsvector` to vector
similarity search. Both results can be merged and deduplicated.
**Deferred to Phase 3** — notes embedding is additive and independent
of the core KB pipeline. Ship KB search first, add note embeddings
after the pipeline is proven.
---
## 8. Per-Channel KB Toggle
### UI: Chat Sidebar
A "Knowledge" button in the chat header or input bar (next to the tools
toggle). Click opens a popup listing available KBs with toggle switches.
```
┌─────────────────────────────────┐
│ 📚 Knowledge Bases │
│ │
│ ☑ Q4 Reports (3 docs) │
│ ☑ Engineering Wiki (12 docs) │
│ ☐ HR Policies (5 docs) │
│ │
│ Manage KBs... │
└─────────────────────────────────┘
```
### State
Active KBs per channel stored in `channel_knowledge_bases`. When a user
toggles a KB on/off for a channel, the frontend calls:
```
PUT /api/v1/channels/:id/knowledge-bases
Body: { kb_ids: ["uuid1", "uuid2"] }
```
The completion handler reads active KBs to populate the system prompt
hint and to scope `kb_search` results.
### Visibility Rules
A user can toggle KBs they have access to:
- **Global KBs** — all users
- **Team KBs** — team members
- **Personal KBs** — owner only
The list endpoint returns all KBs the user can see, with `enabled`
status per channel.
---
## 9. Store Interface
```go
// server/store/interfaces.go — additions
// =========================================
// KNOWLEDGE BASE STORE
// =========================================
type KnowledgeBaseStore interface {
// KB CRUD
Create(ctx context.Context, kb *models.KnowledgeBase) error
GetByID(ctx context.Context, id string) (*models.KnowledgeBase, error)
Update(ctx context.Context, id string, fields map[string]interface{}) error
Delete(ctx context.Context, id string) error
// Scoped listing
ListForUser(ctx context.Context, userID string, teamIDs []string) ([]models.KnowledgeBase, error)
ListGlobal(ctx context.Context) ([]models.KnowledgeBase, error)
ListForTeam(ctx context.Context, teamID string) ([]models.KnowledgeBase, error)
ListPersonal(ctx context.Context, userID string) ([]models.KnowledgeBase, error)
// Documents
CreateDocument(ctx context.Context, doc *models.KBDocument) error
GetDocument(ctx context.Context, id string) (*models.KBDocument, error)
ListDocuments(ctx context.Context, kbID string) ([]models.KBDocument, error)
UpdateDocumentStatus(ctx context.Context, id string, status string, err *string) error
DeleteDocument(ctx context.Context, id string) (*models.KBDocument, error) // returns for storage cleanup
// Chunks
InsertChunks(ctx context.Context, chunks []models.KBChunk) error
DeleteChunksForDocument(ctx context.Context, documentID string) error
SimilaritySearch(ctx context.Context, kbIDs []string, queryVec []float64,
threshold float64, limit int) ([]models.KBSearchResult, error)
// Channel links
SetChannelKBs(ctx context.Context, channelID string, kbIDs []string) error
GetChannelKBs(ctx context.Context, channelID string) ([]models.ChannelKB, error)
GetActiveKBIDs(ctx context.Context, channelID string, userID string,
teamIDs []string) ([]string, error) // enabled + user has access
// Stats
UpdateStats(ctx context.Context, kbID string) error // recount from chunks
}
```
Add `KnowledgeBases KnowledgeBaseStore` to the `Stores` struct.
---
## 10. Models
```go
// server/models/models.go — additions
type KnowledgeBase struct {
ID string `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Description string `json:"description" db:"description"`
Scope string `json:"scope" db:"scope"`
OwnerID *string `json:"owner_id,omitempty" db:"owner_id"`
TeamID *string `json:"team_id,omitempty" db:"team_id"`
EmbeddingConfig map[string]interface{} `json:"embedding_config" db:"embedding_config"`
DocumentCount int `json:"document_count" db:"document_count"`
ChunkCount int `json:"chunk_count" db:"chunk_count"`
TotalBytes int64 `json:"total_bytes" db:"total_bytes"`
Status string `json:"status" db:"status"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
}
type KBDocument struct {
ID string `json:"id" db:"id"`
KBID string `json:"kb_id" db:"kb_id"`
Filename string `json:"filename" db:"filename"`
ContentType string `json:"content_type" db:"content_type"`
SizeBytes int64 `json:"size_bytes" db:"size_bytes"`
StorageKey string `json:"storage_key" db:"storage_key"`
ExtractedText *string `json:"extracted_text,omitempty" db:"extracted_text"`
ChunkCount int `json:"chunk_count" db:"chunk_count"`
Status string `json:"status" db:"status"`
Error *string `json:"error,omitempty" db:"error"`
UploadedBy string `json:"uploaded_by" db:"uploaded_by"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
}
type KBChunk struct {
ID string `json:"id" db:"id"`
KBID string `json:"kb_id" db:"kb_id"`
DocumentID string `json:"document_id" db:"document_id"`
ChunkIndex int `json:"chunk_index" db:"chunk_index"`
Content string `json:"content" db:"content"`
TokenCount int `json:"token_count" db:"token_count"`
Embedding []float64 `json:"-" db:"embedding"` // not serialized to API
Metadata map[string]interface{} `json:"metadata" db:"metadata"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
}
type KBSearchResult struct {
Content string `json:"content"`
Filename string `json:"source"`
KBName string `json:"kb"`
Similarity float64 `json:"similarity"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
type ChannelKB struct {
KBID string `json:"kb_id" db:"kb_id"`
KBName string `json:"kb_name" db:"name"`
Enabled bool `json:"enabled" db:"enabled"`
DocCount int `json:"document_count" db:"document_count"`
}
```
---
## 11. File Structure
### New Go Packages
```
server/
├── knowledge/ # NEW — chunking + embedding logic
│ ├── chunker.go # recursive text splitter
│ ├── chunker_test.go
│ ├── embedder.go # batch embed via role resolver
│ └── ingest.go # orchestrator: extract → chunk → embed → store
├── handlers/
│ ├── knowledge_bases.go # NEW — KB CRUD + document upload + search
│ └── ... (existing)
├── store/
│ └── postgres/
│ ├── knowledge_bases.go # NEW — KnowledgeBaseStore impl
│ └── ... (existing)
├── tools/
│ ├── kbsearch.go # NEW — kb_search tool
│ └── ... (existing)
├── database/
│ └── migrations/
│ └── 008_knowledge_bases.sql # NEW
└── models/
└── models.go # additions (KB types)
```
### New/Modified Frontend Files
```
src/js/
├── ui-admin.js # MODIFIED — KB management section in admin panel
├── ui-primitives.js # MODIFIED — embedding dropdown fix, manual entry
├── chat.js # MODIFIED — KB toggle popup, system prompt hint
├── api.js # MODIFIED — KB API methods
└── settings-handlers.js # MODIFIED — personal KB management
```
### Go Dependencies
```
go get github.com/pgvector/pgvector-go # pgvector type support for lib/pq
```
---
## 12. Admin Panel Integration
KBs live under **AI → Knowledge Bases** in the admin panel (v0.13.0).
### Admin View
```
┌──────────────────────────────────────────────────────┐
│ AI → Knowledge Bases │
│ │
│ [+ New Knowledge Base] │
│ │
│ ┌────────────────────────────────────────────────┐ │
│ │ Q4 Reports global 3 docs ready │ │
│ │ Engineering Wiki global 12 docs ready │ │
│ │ Onboarding Materials team:Eng 5 docs ready │ │
│ └────────────────────────────────────────────────┘ │
│ │
│ ── Selected: Q4 Reports ────────────────────────── │
│ │
│ Name: [Q4 Reports ] │
│ Description: [Quarterly financial reports ] │
│ Scope: [Global ▾] Team: [— ▾] │
│ │
│ Embedding: OpenAI / text-embedding-3-small │
│ Chunk size: [1000] Overlap: [200] │
│ │
│ Documents: │
│ ┌────────────────────────────────────────────────┐ │
│ │ quarterly-report-q4.pdf 2.1 MB 42 chunks ✓ │ │
│ │ revenue-breakdown.xlsx 340 KB 12 chunks ✓ │ │
│ │ market-analysis.docx 1.8 MB processing... │ │
│ └────────────────────────────────────────────────┘ │
│ [📎 Upload Documents] [🔄 Rebuild All] [🗑 Delete] │
└──────────────────────────────────────────────────────┘
```
### Personal KB (Settings)
BYOK users see a "Knowledge Bases" section in Settings → Model Roles
(or a new Settings tab). Same UI, scoped to personal KBs using the
user's embedding role override.
### Team KB (Team Management)
Team admins see a "Knowledge Bases" tab in the team management modal.
Same UI pattern, scoped to the team.
---
## 13. Phased Delivery
### Phase 1: Schema + Chunker + Embed Fix
**Goal:** Database ready, chunking works, embedding dropdown fixed.
- [ ] `008_knowledge_bases.sql` migration (pgvector + tables)
- [ ] `go get pgvector-go`, update `go.mod`
- [ ] `knowledge/chunker.go` — recursive text splitter + tests
- [ ] Embedding dropdown fix in `ui-primitives.js` (tolerant filter + manual entry)
- [ ] `KnowledgeBaseStore` interface + postgres implementation (CRUD only, no vector ops)
- [ ] `KnowledgeBase`, `KBDocument`, `KBChunk` model types
- [ ] Wire `Stores.KnowledgeBases` in main.go
### Phase 2: Ingestion Pipeline
**Goal:** Documents can be uploaded, chunked, and embedded end-to-end.
- [ ] `knowledge/embedder.go` — batch embed via role resolver
- [ ] `knowledge/ingest.go` — orchestrator (extract → chunk → embed → store)
- [ ] `handlers/knowledge_bases.go` — KB CRUD endpoints
- [ ] Document upload with async ingestion goroutine
- [ ] Status polling endpoint
- [ ] `SimilaritySearch()` in store (pgvector query)
- [ ] Admin panel: KB management section (AI → Knowledge Bases)
- [ ] IVFFlat index creation (manual or after first batch)
### Phase 3: kb_search Tool + Channel Integration
**Goal:** LLMs can search KBs during conversations.
- [ ] `tools/kbsearch.go` — kb_search tool (late registration pattern)
- [ ] Per-channel KB toggle (channel_knowledge_bases)
- [ ] System prompt KB hint injection in completion handler
- [ ] KB toggle popup in chat UI
- [ ] Frontend: channel KB management
- [ ] Category "knowledge" in tools toggle menu
### Phase 4: Polish + Notes
**Goal:** Team/personal scoping works, notes get vectors.
- [ ] Team KB endpoints + team management UI tab
- [ ] Personal KB endpoints + settings UI
- [ ] KB rebuild endpoint (re-chunk + re-embed)
- [ ] Notes embedding column + semantic note search
- [ ] Audit logging for KB operations
- [ ] Usage tracking for embedding calls (role = 'embedding')
- [ ] Integration tests
---
## 14. Risks + Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| pgvector not installed on PG host | Migration fails, blocks deploy | Clear error message in migration; add to deployment docs/Dockerfile |
| Embedding provider rate limits | Ingestion stalls on large uploads | Semaphore + exponential backoff; batch size tuning |
| Dimension mismatch (model change) | Search returns garbage | Snapshot model in `embedding_config`; warn if role model ≠ KB model |
| Large documents (100+ pages) | Memory pressure during chunking | Stream text extraction; chunk incrementally |
| Vector storage size | DB bloat with many KBs | Monitor with stats endpoint; future: HNSW index for better perf |
| Embedding dropdown broken | Can't configure embedding role at all | Phase 1 priority — ship fix before KB features |
---
## 15. Future (Not v0.14.0)
- **Auto-injection:** Pre-pend top-K results to system prompt (v0.15.x, needs compaction for context budget)
- **Hybrid search:** Combine vector similarity with full-text `tsvector` search, re-rank
- **Semantic chunking:** Use embedding distance to detect topic boundaries
- **HNSW index:** Better query performance than IVFFlat for large datasets
- **Web scraping source:** Ingest URLs as KB documents (natural extension of url_fetch)
- **Scheduled re-indexing:** Periodic rebuild when source documents are updated
- **KB sharing:** Cross-team KB access grants

795
docs/DESIGN-0.16.0.md
View File

@@ -1,795 +0,0 @@
# DESIGN-0.16.0 — User Groups + Resource Grants
## Overview
The missing access-control primitive. Teams define organizational structure
(horizontal visibility). Roles define vertical permissions (admin/user).
**Groups** are pure access-control lists that decouple resource access from
team membership.
Today, resource visibility follows a rigid scope model: `global` (everyone),
`team` (team members), `personal` (owner only). This breaks down when:
- A KB should be visible to members from two different teams.
- A Persona should be accessible by a cross-functional working group.
- A future Project needs participants from multiple teams.
Groups solve this by adding a fourth access path — grant-by-group — without
changing the existing scope model. Existing `global/team/personal` access
continues to work unchanged. Groups are additive.
Depends on: teams (v0.10.0), knowledge bases (v0.14.0).
**Design principle: don't break the scope model.** Groups extend it. Every
existing query that filters by `scope + owner_id + team_id` remains valid.
Group membership is a parallel access path checked alongside scope, not a
replacement for it.
---
## 1. Schema
Migration: `010_groups.sql`
### 1.1 Groups Table
```sql
-- =========================================
-- USER GROUPS
-- =========================================
CREATE TABLE IF NOT EXISTS groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
description TEXT NOT NULL DEFAULT '',
scope VARCHAR(20) NOT NULL DEFAULT 'global'
CHECK (scope IN ('global', 'team')),
team_id UUID REFERENCES teams(id) ON DELETE CASCADE,
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Global groups: team_id must be NULL
-- Team groups: team_id must be set
CONSTRAINT groups_scope_team CHECK (
(scope = 'global' AND team_id IS NULL) OR
(scope = 'team' AND team_id IS NOT NULL)
)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_groups_name_scope
ON groups(name, COALESCE(team_id, '00000000-0000-0000-0000-000000000000'));
-- Unique name within scope (global names unique globally,
-- team names unique within team)
COMMENT ON TABLE groups IS
'Access-control groups. Decouple resource visibility from team membership.';
COMMENT ON COLUMN groups.scope IS
'global: admin-managed, can span teams. team: team-admin-managed, team-internal.';
```
### 1.2 Group Members Table
```sql
CREATE TABLE IF NOT EXISTS group_members (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
added_by UUID NOT NULL REFERENCES users(id),
added_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(group_id, user_id)
);
CREATE INDEX IF NOT EXISTS idx_group_members_user ON group_members(user_id);
CREATE INDEX IF NOT EXISTS idx_group_members_group ON group_members(group_id);
```
### 1.3 Resource Grants Table
```sql
-- =========================================
-- RESOURCE GRANTS
-- =========================================
CREATE TABLE IF NOT EXISTS resource_grants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
resource_type VARCHAR(30) NOT NULL
CHECK (resource_type IN ('persona', 'knowledge_base')),
resource_id UUID NOT NULL,
grant_scope VARCHAR(20) NOT NULL DEFAULT 'team_only'
CHECK (grant_scope IN ('team_only', 'global', 'groups')),
granted_groups UUID[] NOT NULL DEFAULT '{}',
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Only one grant row per resource
UNIQUE(resource_type, resource_id)
);
CREATE INDEX IF NOT EXISTS idx_resource_grants_resource
ON resource_grants(resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_resource_grants_groups
ON resource_grants USING gin(granted_groups);
-- GIN index for UUID[] containment queries (&&, @>)
COMMENT ON TABLE resource_grants IS
'Controls cross-team access to resources (personas, KBs, future: projects).';
COMMENT ON COLUMN resource_grants.grant_scope IS
'team_only: existing team scope behavior. global: all authenticated users. groups: specific group list.';
COMMENT ON COLUMN resource_grants.granted_groups IS
'UUID array of group IDs. Only meaningful when grant_scope = groups.';
```
### 1.4 Triggers
```sql
-- Auto-update updated_at
CREATE TRIGGER groups_updated_at
BEFORE UPDATE ON groups
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
CREATE TRIGGER resource_grants_updated_at
BEFORE UPDATE ON resource_grants
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
```
### Design Decision: UUID[] vs Junction Table
The roadmap specifies `granted_groups UUID[]` on `resource_grants`. This is
the right call for this use case:
- A resource has exactly one grant row (UNIQUE constraint).
- The group list is always read/written as a unit (replace-all semantics).
- Postgres `UUID[] && ARRAY[...]::uuid[]` with a GIN index is fast for
"does this resource grant overlap with the user's groups?" queries.
- Avoids a many-to-many junction table that would complicate the common
path (checking access) for marginal normalization benefit.
A junction table (`resource_grant_groups`) would be warranted if we needed
per-group config on grants (e.g., read-only vs read-write per group). We
don't — the grant is binary (access or no access). If that changes, we
can migrate the array to a junction table later.
---
## 2. Models
### 2.1 Group Models
```go
// ── Group Constants ────────────────────────
const (
GroupScopeGlobal = "global"
GroupScopeTeam = "team"
)
// ── Groups ─────────────────────────────────
type Group struct {
ID string `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Description string `json:"description" db:"description"`
Scope string `json:"scope" db:"scope"`
TeamID *string `json:"team_id,omitempty" db:"team_id"`
CreatedBy string `json:"created_by" db:"created_by"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
// Computed fields (not stored in groups table)
MemberCount int `json:"member_count,omitempty"`
TeamName string `json:"team_name,omitempty"`
}
type GroupPatch struct {
Name *string `json:"name,omitempty"`
Description *string `json:"description,omitempty"`
}
type GroupMember struct {
ID string `json:"id" db:"id"`
GroupID string `json:"group_id" db:"group_id"`
UserID string `json:"user_id" db:"user_id"`
AddedBy string `json:"added_by" db:"added_by"`
AddedAt string `json:"added_at" db:"added_at"`
// Joined from users table
Username string `json:"username,omitempty"`
DisplayName string `json:"display_name,omitempty"`
Email string `json:"email,omitempty"`
}
```
### 2.2 Resource Grant Models
```go
// ── Resource Grant Constants ───────────────
const (
ResourceTypePersona = "persona"
ResourceTypeKnowledgeBase = "knowledge_base"
// Future: "project"
GrantScopeTeamOnly = "team_only"
GrantScopeGlobal = "global"
GrantScopeGroups = "groups"
)
// ResourceGrant controls cross-team access to a resource.
type ResourceGrant struct {
ID string `json:"id" db:"id"`
ResourceType string `json:"resource_type" db:"resource_type"`
ResourceID string `json:"resource_id" db:"resource_id"`
GrantScope string `json:"grant_scope" db:"grant_scope"`
GrantedGroups []string `json:"granted_groups" db:"granted_groups"` // UUID[]
CreatedBy string `json:"created_by" db:"created_by"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
}
```
---
## 3. Store Interfaces
### 3.1 GroupStore
```go
// =========================================
// GROUP STORE
// =========================================
type GroupStore interface {
// CRUD
Create(ctx context.Context, g *models.Group) error
GetByID(ctx context.Context, id string) (*models.Group, error)
Update(ctx context.Context, id string, patch models.GroupPatch) error
Delete(ctx context.Context, id string) error
// Listing
ListAll(ctx context.Context) ([]models.Group, error) // admin view
ListForTeam(ctx context.Context, teamID string) ([]models.Group, error)
ListForUser(ctx context.Context, userID string) ([]models.Group, error) // groups I'm in
// Members
AddMember(ctx context.Context, groupID, userID, addedBy string) error
RemoveMember(ctx context.Context, groupID, userID string) error
ListMembers(ctx context.Context, groupID string) ([]models.GroupMember, error)
IsMember(ctx context.Context, groupID, userID string) (bool, error)
// Bulk lookups (for access resolution)
GetUserGroupIDs(ctx context.Context, userID string) ([]string, error)
}
```
### 3.2 ResourceGrantStore
```go
// =========================================
// RESOURCE GRANT STORE
// =========================================
type ResourceGrantStore interface {
// Upsert: one grant row per resource (replace semantics)
Set(ctx context.Context, grant *models.ResourceGrant) error
Get(ctx context.Context, resourceType, resourceID string) (*models.ResourceGrant, error)
Delete(ctx context.Context, resourceType, resourceID string) error
// Access check: does user have group-based access to this resource?
UserHasGroupAccess(ctx context.Context, userID, resourceType, resourceID string) (bool, error)
}
```
### 3.3 Wire into Stores Bundle
```go
type Stores struct {
// ... existing fields ...
Groups GroupStore
ResourceGrants ResourceGrantStore
}
```
---
## 4. Access Resolution
### 4.1 The Problem
Today, access checks look like this (Persona example):
```sql
-- PersonaStore.ListForUser
WHERE scope = 'global'
OR (scope = 'personal' AND created_by = $1)
OR (scope = 'team' AND owner_id IN (SELECT team_id FROM team_members WHERE user_id = $1))
OR (scope = 'personal' AND is_shared = true)
```
KB access follows the same pattern in `KnowledgeBaseStore.ListForUser` and
the handler's `userCanAccess()` method.
Groups add a fourth access path: "I'm in a group that has been granted
access to this resource."
### 4.2 Design: Parallel Check, Not Query Rewrite
We could embed the group check into every `ListForUser` query. That would
mean every listing query joins against `resource_grants` and `group_members`,
adding complexity to every scope-aware query in the system.
Instead: **keep existing scope queries as-is, add group-granted resources
as a separate UNION.** This is cleaner, backward-compatible, and doesn't
slow down the common path for users without group-based access.
### 4.3 Updated Persona Listing
```go
// PersonaStore.ListForUser — updated
func (s *PersonaStore) ListForUser(ctx context.Context, userID string) ([]models.Persona, error) {
rows, err := DB.QueryContext(ctx, fmt.Sprintf(`
-- Existing scope-based access (unchanged)
SELECT %s FROM personas WHERE is_active = true AND (
scope = 'global'
OR (scope = 'personal' AND created_by = $1)
OR (scope = 'team' AND owner_id IN (
SELECT team_id FROM team_members WHERE user_id = $1
))
OR (scope = 'personal' AND is_shared = true)
)
UNION
-- Group-based access (new)
SELECT %s FROM personas p
WHERE p.is_active = true
AND p.id IN (
SELECT rg.resource_id FROM resource_grants rg
WHERE rg.resource_type = 'persona'
AND (
rg.grant_scope = 'global'
OR (rg.grant_scope = 'groups' AND rg.granted_groups && (
SELECT ARRAY_AGG(gm.group_id) FROM group_members gm
WHERE gm.user_id = $1
)::uuid[])
)
)
ORDER BY scope, name
`, personaCols, personaCols), userID)
// ...
}
```
The `&&` operator is the Postgres array overlap operator — returns true if
the two arrays share any element. The GIN index on `granted_groups` makes
this efficient.
### 4.4 Updated KB Listing
Same pattern applied to `KnowledgeBaseStore.ListForUser`:
```go
func (s *KnowledgeBaseStore) ListForUser(ctx context.Context, userID string, teamIDs []string) ([]models.KnowledgeBase, error) {
// ... existing scope-based query ...
// Add UNION for group-granted KBs
q += `
UNION
SELECT id, name, description, scope, owner_id, team_id, embedding_config,
document_count, chunk_count, total_bytes, status, created_at, updated_at
FROM knowledge_bases kb
WHERE kb.id IN (
SELECT rg.resource_id FROM resource_grants rg
WHERE rg.resource_type = 'knowledge_base'
AND (
rg.grant_scope = 'global'
OR (rg.grant_scope = 'groups' AND rg.granted_groups && (
SELECT ARRAY_AGG(gm.group_id) FROM group_members gm
WHERE gm.user_id = $1
)::uuid[])
)
)`
q += ` ORDER BY name`
return queryKBs(ctx, q, args...)
}
```
### 4.5 Updated UserCanAccess
The KB handler's `userCanAccess()` method also needs updating:
```go
func (h *KnowledgeBaseHandler) userCanAccess(kb *models.KnowledgeBase, userID string, teamIDs []string) bool {
// Existing scope-based check (unchanged)
switch kb.Scope {
case "global":
return true
case "personal":
if kb.OwnerID != nil && *kb.OwnerID == userID {
return true
}
case "team":
if kb.TeamID != nil {
for _, tid := range teamIDs {
if tid == *kb.TeamID {
return true
}
}
}
}
// New: check group-based grants
hasAccess, _ := h.stores.ResourceGrants.UserHasGroupAccess(
context.Background(), userID, models.ResourceTypeKnowledgeBase, kb.ID)
return hasAccess
}
```
The `PersonaStore.UserCanAccess` method gets the same treatment.
### 4.6 GetActiveKBIDs (Channel KB Access)
The `GetActiveKBIDs` query (used by the completion handler to determine which
KBs to search) also needs the group path:
```sql
-- Add to the WHERE clause:
OR kb.id IN (
SELECT rg.resource_id FROM resource_grants rg
WHERE rg.resource_type = 'knowledge_base'
AND (
rg.grant_scope = 'global'
OR (rg.grant_scope = 'groups' AND rg.granted_groups && (
SELECT ARRAY_AGG(gm.group_id) FROM group_members gm
WHERE gm.user_id = $2
)::uuid[])
)
)
```
### 4.7 Performance Note: User Group IDs Cache
The `ARRAY_AGG(gm.group_id)` subquery appears in every access check. For
hot paths (listing, access checks), we can pre-fetch the user's group IDs
once per request and pass them through:
```go
// In middleware or handler setup:
groupIDs, _ := stores.Groups.GetUserGroupIDs(ctx, userID)
// Then in queries, pass as a parameter instead of subquery:
// ... AND rg.granted_groups && $N::uuid[]
```
This trades a subquery for a pre-fetched array parameter. Worth doing in
phase 2 if access checks show up in profiling. For v0.16.0, the subquery
approach is simpler and correct.
---
## 5. Handlers
### 5.1 GroupHandler
```go
type GroupHandler struct {
stores store.Stores
}
func NewGroupHandler(s store.Stores) *GroupHandler {
return &GroupHandler{stores: s}
}
```
**Admin endpoints (global groups):**
| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/admin/groups` | ListAllGroups | admin |
| POST | `/admin/groups` | CreateGlobalGroup | admin |
| PUT | `/admin/groups/:id` | UpdateGroup | admin |
| DELETE | `/admin/groups/:id` | DeleteGroup | admin |
| GET | `/admin/groups/:id/members` | ListGroupMembers | admin |
| POST | `/admin/groups/:id/members` | AddGroupMember | admin |
| DELETE | `/admin/groups/:id/members/:userId` | RemoveGroupMember | admin |
**Team endpoints (team-scoped groups):**
| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/teams/:teamId/groups` | ListTeamGroups | team member |
| POST | `/teams/:teamId/groups` | CreateTeamGroup | team admin |
| PUT | `/teams/:teamId/groups/:id` | UpdateTeamGroup | team admin |
| DELETE | `/teams/:teamId/groups/:id` | DeleteTeamGroup | team admin |
| GET | `/teams/:teamId/groups/:id/members` | ListTeamGroupMembers | team member |
| POST | `/teams/:teamId/groups/:id/members` | AddTeamGroupMember | team admin |
| DELETE | `/teams/:teamId/groups/:id/members/:userId` | RemoveTeamGroupMember | team admin |
**User endpoint:**
| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/groups/mine` | MyGroups | authenticated |
### 5.2 Resource Grant Endpoints
Grant management is inline on the resource endpoints, not a separate handler.
**On Personas:**
| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/presets/:id/grants` | GetPersonaGrants | owner or admin |
| PUT | `/presets/:id/grants` | SetPersonaGrants | owner or admin |
**On Knowledge Bases:**
| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/knowledge-bases/:id/grants` | GetKBGrants | owner or admin |
| PUT | `/knowledge-bases/:id/grants` | SetKBGrants | owner or admin |
**Request/response format:**
```json
// PUT /api/v1/presets/:id/grants
{
"grant_scope": "groups",
"granted_groups": ["uuid-1", "uuid-2"]
}
// GET /api/v1/presets/:id/grants
{
"grant_scope": "groups",
"granted_groups": [
{ "id": "uuid-1", "name": "Engineering" },
{ "id": "uuid-2", "name": "Design" }
]
}
```
The GET response enriches group UUIDs with names for display. The PUT
accepts bare UUIDs.
---
## 6. Routes (main.go)
```go
// ── Groups ──────────────────────────────
// User: my groups
protected.GET("/groups/mine", groupH.MyGroups)
// Team-scoped groups
teamScoped.GET("/groups", groupH.ListTeamGroups)
teamScoped.POST("/groups", groupH.CreateTeamGroup) // RequireTeamAdmin
teamScoped.PUT("/groups/:groupId", groupH.UpdateTeamGroup) // RequireTeamAdmin
teamScoped.DELETE("/groups/:groupId", groupH.DeleteTeamGroup) // RequireTeamAdmin
teamScoped.GET("/groups/:groupId/members", groupH.ListTeamGroupMembers)
teamScoped.POST("/groups/:groupId/members", groupH.AddTeamGroupMember) // RequireTeamAdmin
teamScoped.DELETE("/groups/:groupId/members/:userId", groupH.RemoveTeamGroupMember) // RequireTeamAdmin
// Admin: global groups
admin.GET("/groups", groupH.ListAllGroups)
admin.POST("/groups", groupH.CreateGlobalGroup)
admin.PUT("/groups/:id", groupH.UpdateGroup)
admin.DELETE("/groups/:id", groupH.DeleteGroup)
admin.GET("/groups/:id/members", groupH.ListGroupMembers)
admin.POST("/groups/:id/members", groupH.AddGroupMember)
admin.DELETE("/groups/:id/members/:userId", groupH.RemoveGroupMember)
// ── Resource Grants ─────────────────────
// Persona grants (add to existing preset routes)
protected.GET("/presets/:id/grants", personaH.GetGrants)
protected.PUT("/presets/:id/grants", personaH.SetGrants)
// KB grants (add to existing KB routes)
protected.GET("/knowledge-bases/:id/grants", kbH.GetGrants)
protected.PUT("/knowledge-bases/:id/grants", kbH.SetGrants)
```
---
## 7. Audit Logging
All group and grant operations log to the existing audit system:
| Action | Resource Type | Details |
|--------|--------------|---------|
| `group.create` | `group` | name, scope, team_id |
| `group.update` | `group` | changed fields |
| `group.delete` | `group` | name |
| `group.member.add` | `group` | user_id |
| `group.member.remove` | `group` | user_id |
| `resource_grant.set` | `persona` / `knowledge_base` | grant_scope, group count |
| `resource_grant.delete` | `persona` / `knowledge_base` | — |
Uses the existing `AuditStore.Log` method — no new infrastructure.
---
## 8. Frontend
### 8.1 Admin Panel: Groups Section
New tab in the admin panel (alongside Users, Teams, Models, Settings, etc.):
**Groups list view:**
- Table: Name, Scope, Team (if team-scoped), Members count, Actions
- "Create Group" button → modal with name, description, scope selector
- Click row → member management (add/remove users, search by username)
**Team admin panel:**
- New "Groups" tab under team settings
- Same list/CRUD pattern, scoped to team
### 8.2 Grant Picker Component
Reusable component for Persona and KB create/edit forms:
```
┌─ Access ──────────────────────────────────┐
│ ○ Team Only (default) │
│ ○ All Users │
│ ○ Specific Groups │
│ ┌────────────────────────────────────┐ │
│ │ [x] Engineering │ │
│ │ [x] Design │ │
│ │ [ ] Marketing │ │
│ │ [ ] Leadership │ │
│ └────────────────────────────────────┘ │
└────────────────────────────────────────────┘
```
- Only shown for team-scoped and global-scoped resources (personal resources
are always owner-only).
- "Team Only" = existing behavior (no `resource_grants` row, or row with
`grant_scope = 'team_only'`).
- "All Users" = `grant_scope = 'global'`.
- "Specific Groups" = `grant_scope = 'groups'` + multi-select group picker.
- Group list loaded from `/api/v1/groups/mine` (user's groups) for team
admins, `/api/v1/admin/groups` for system admins.
### 8.3 User's Group Badge
The model selector and KB list already show scope badges (`global`, `team`,
`personal`). For group-granted resources, show a "group" badge to distinguish
from team-native access.
---
## 9. Backward Compatibility
### 9.1 No Breaking Changes
- All existing scope-based queries remain valid.
- Resources without a `resource_grants` row behave exactly as before.
- The `resource_grants` table is opt-in: only populated when an admin or
team admin explicitly sets grants on a resource.
- `persona_grants` (tool/KB/API grants ON a persona) is unrelated to
`resource_grants` (access grants TO a persona). Different concepts,
different tables. No rename needed.
### 9.2 Migration Safety
- New tables only (no ALTER on existing tables).
- No data migration required.
- Rollback: drop the three new tables.
### 9.3 Naming Clarity
To avoid confusion between the existing `persona_grants` table (which
controls what tools/KBs a Persona has access TO) and the new
`resource_grants` table (which controls which users have access TO the
Persona):
| Table | Purpose | Example |
|-------|---------|---------|
| `persona_grants` | What a Persona can do (tools, KBs, APIs) | "CodeBot can use calculator" |
| `resource_grants` | Who can use a resource (users via groups) | "Engineering group can use CodeBot" |
The naming is intentionally distinct. No rename of `persona_grants` is needed.
---
## 10. Implementation Phases
### Phase 1: Groups Entity (backend)
- [ ] Migration `010_groups.sql`
- [ ] Models: `Group`, `GroupPatch`, `GroupMember`
- [ ] Store: `GroupStore` interface + Postgres implementation
- [ ] Handler: `GroupHandler` with admin + team admin + user endpoints
- [ ] Routes: wire into `main.go`
- [ ] Audit logging for group operations
- [ ] Integration tests: CRUD, membership, scoping
### Phase 2: Resource Grants (backend)
- [ ] Models: `ResourceGrant` + constants
- [ ] Store: `ResourceGrantStore` interface + Postgres implementation
- [ ] Grant endpoints on PersonaHandler and KBHandler
- [ ] Update `PersonaStore.ListForUser` with UNION for group access
- [ ] Update `PersonaStore.UserCanAccess` with group check
- [ ] Update `KnowledgeBaseStore.ListForUser` with UNION for group access
- [ ] Update `KBHandler.userCanAccess` with group check
- [ ] Update `KnowledgeBaseStore.GetActiveKBIDs` with group path
- [ ] Audit logging for grant operations
- [ ] Integration tests: grant CRUD, access resolution, cross-team visibility
### Phase 3: Frontend
- [ ] Admin panel: Groups section (CRUD, member management)
- [ ] Team admin panel: Groups tab
- [ ] Grant picker component (shared between Persona and KB forms)
- [ ] `/groups/mine` endpoint for user's group list
- [ ] "Group" scope badge on model selector / KB list
- [ ] Update Persona create/edit form with grant picker
- [ ] Update KB create/edit form with grant picker
---
## 11. Testing Plan
### Integration Tests (extend `handlers/integration_test.go`)
**Group lifecycle:**
1. Admin creates global group → verify in list
2. Admin adds users to group → verify membership
3. Team admin creates team group → verify scoping (not visible to other teams)
4. Remove member → verify no longer in group
5. Delete group → verify CASCADE cleans up members
**Resource grants:**
1. Team admin creates a KB with `grant_scope = 'groups'`, grants to a group
2. User in the group can see the KB in `ListForUser` → verify access
3. User NOT in the group cannot see the KB → verify denied
4. User in group can search the KB via `kb_search` tool → verify access
5. Remove user from group → verify KB no longer visible
6. Change grant from `groups` to `global` → verify all users can access
7. Delete grant → verify KB reverts to team-only visibility
**Backward compatibility:**
1. Resources with no `resource_grants` row → same behavior as before
2. Existing scope queries → unchanged results
3. Personal resources → never grant-accessible (grant picker hidden)
### Edge Cases
- User belongs to 0 groups → `ARRAY_AGG` returns NULL → no match (safe)
- Group with 0 members → grant exists but no one matches (correct)
- Delete a group that's referenced in `granted_groups` → array still contains
the UUID but no `group_members` rows match → effectively revoked (correct).
Cleanup: periodic job or ON DELETE trigger to scrub stale UUIDs from arrays.
For v0.16.0, stale UUIDs are harmless (they just don't match anything).
---
## 12. Future Considerations
### v0.17.0 — Persona-KB Binding
Resource grants enable the access control layer for Persona-KB binding.
When a Persona has bound KBs, the Persona's grants implicitly grant search
access to those KBs. The `resource_grants` table already supports this —
the Persona grant controls who can use the Persona, and Persona-KB binding
controls which KBs are searchable through it.
### v0.19.0 — Projects
Projects will use the same `resource_grants` pattern:
`resource_type = 'project'`. No schema changes needed.
### v0.24.0 — Full RBAC
OIDC claim → group mapping will auto-sync external IdP groups into the
`groups` table. The infrastructure built here becomes the foundation for
enterprise RBAC.
### Stale Group UUID Cleanup
When a group is deleted, its UUID may linger in `resource_grants.granted_groups`
arrays. This is harmless (dead UUIDs never match any `group_members` rows)
but untidy. Options for a future cleanup:
- Trigger on `groups` DELETE that scrubs the UUID from all `granted_groups` arrays.
- Periodic background job.
- Cleanup on next grant SET (when the admin edits the grant).
For v0.16.0, we'll add a note in the delete handler response if the group
is referenced in any grants, warning the admin.

184
docs/DESIGN-0.17.0.md
View File

@@ -1,184 +0,0 @@
# DESIGN-0.17.0 — Persona-KB Binding + Enterprise KB Mode
## Overview
Personas become **gateways** to knowledge. In enterprise deployments,
users don't interact with KBs directly — they talk to Personas that
have KBs attached. This is the core differentiator for enterprise use.
Depends on: knowledge bases (v0.14.0), user groups (v0.16.0).
**Design principle: Personas carry context, not users.** When a user
selects a Persona with bound KBs, the KB context flows automatically —
no manual KB selection, no toggle management, no confusion about which
KBs are in scope. Admins curate the KB↔Persona relationships; users
just pick a Persona.
---
## 1. Schema
### `persona_knowledge_bases` (new join table)
```sql
CREATE TABLE persona_knowledge_bases (
persona_id UUID NOT NULL REFERENCES personas(id) ON DELETE CASCADE,
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
auto_search BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (persona_id, kb_id)
);
```
`auto_search` controls whether the KB is searched automatically on every
message (top-K results prepended to context) or only via explicit
`kb_search` tool calls. Default `false` = tool-only.
### `knowledge_bases.discoverable` (new column)
```sql
ALTER TABLE knowledge_bases ADD COLUMN discoverable BOOLEAN NOT NULL DEFAULT true;
```
When `false`, the KB does not appear in user-facing listings
(`ListDiscoverableKBs`). It remains searchable through Persona bindings.
### `platform_policies.kb_direct_access` (new row)
Seeded as `'true'` (permissive default). When set to `'false'`, the
channel KB popup is hidden — users access KBs exclusively through
Persona bindings.
---
## 2. Store Layer
### `PersonaStore` additions
- `SetKBs(ctx, personaID, kbIDs, autoSearch)` — UPSERT join table
- `GetKBs(ctx, personaID)` — returns `[]PersonaKB` with KB metadata
### `KnowledgeBaseStore` additions
- `ListDiscoverable(ctx, userID, teamIDs)` — KBs where `discoverable=true`
and user has access via ownership, team, or global scope
- `SetDiscoverable(ctx, kbID, discoverable)` — toggle visibility
- `UpdateDocumentStorageKey(ctx, docID, key)` — moved from raw SQL
---
## 3. Completion Pipeline
### KB scoping in `BuildKBHint`
When a Persona has bound KBs, `BuildKBHint` merges them with any
channel-attached KBs:
1. Load channel KBs (existing path)
2. Load Persona KBs via `PersonaStore.GetKBs()`
3. Union the two sets (deduplicated by KB ID)
4. Build the hint string with listing
### Persona ID threading
The `personaID` is threaded through the completion handler methods:
`Complete()` → `streamCompletion(…, personaID)` / `syncCompletion(…, personaID)`
→ `loadConversation(…, personaID)`.
The `ExecutionContext` for tool calls carries the Persona ID so
`kb_search` can auto-include Persona-bound KBs in its search scope.
---
## 4. Role Fallback Alerts (issue #69)
When a role's primary provider fails:
1. **Attempt fallback** — existing behavior, unchanged.
2. **Emit event** — `role.fallback` on EventBus (DirToClient).
3. **Cooldown** — 5-minute per-role TTL prevents flooding.
4. **Admin toast** — frontend listens for `role.fallback` events,
shows warning/error toast to admin users only.
### Resolver changes
```go
type Resolver struct {
stores store.Stores
vault *crypto.KeyResolver
bus *events.Bus
mu sync.Mutex
cooldown map[string]time.Time // role → last alert time
}
```
`WithBus(bus)` builder method. Nil-safe — if no bus is attached,
alerts are suppressed (log only).
---
## 5. Security Fixes
### ResolvePreset bypass
`ResolvePreset()` in `presets.go` used raw SQL that skipped the
`resource_grants` table added in v0.16.0. A Persona accessible only
via group grant would fail at completion time. Fixed to use
`PersonaStore.GetByID()` + `UserCanAccess()`.
### KB create scope authorization
`POST /api/v1/knowledge-bases` now enforces:
- `scope=global` → requires admin role
- `scope=team` → requires team admin role for the target team
- `scope=personal` → open to all authenticated users
---
## 6. Frontend
### KB Picker (`persona-kb.js`)
Standalone component: `renderPersonaKBPicker(container, opts)` → control
object with `getValues()`, `setValues()`, `clear()`, `refresh()`.
Options: `scope` ('admin'|'team'|'personal'), `teamId`, `prefix`.
Shows checkboxes for available KBs with doc/chunk counts. Selected KBs
show an "Auto-inject" toggle for the `auto_search` flag.
### Integration points
- Admin preset form: KB picker below the form, load/save bindings on
edit/create via `/api/v1/admin/presets/:id/knowledge-bases`.
- Team preset form: same pattern via team API prefix.
- User preset form: shows discoverable KBs only.
- Admin settings: `kb_direct_access` toggle checkbox.
### Fallback alert
Admin users receive `UI.toast()` notifications when `role.fallback`
events arrive via WebSocket.
---
## 7. API Endpoints (new)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/presets/:id/knowledge-bases` | List KBs bound to a persona |
| PUT | `/api/v1/presets/:id/knowledge-bases` | Set persona-KB bindings |
| GET | `/api/v1/admin/presets/:id/knowledge-bases` | Admin: list persona KBs |
| PUT | `/api/v1/admin/presets/:id/knowledge-bases` | Admin: set persona-KB bindings |
| GET | `/api/v1/knowledge-bases-discoverable` | List discoverable KBs for user |
| PUT | `/api/v1/knowledge-bases/:id/discoverable` | Toggle KB discoverability |
---
## 8. Migration
Single migration file: `002_v017_persona_kb.sql`
- Creates `persona_knowledge_bases` table
- Adds `discoverable` column to `knowledge_bases`
- Seeds `kb_direct_access` platform policy (default `'true'`)

1262
docs/ROADMAP.md
View File

File diff suppressed because it is too large Load Diff

10
docs/CHANGES-0.18.0-phase1.md → docs/archive/CHANGES-0.18.0-phase1.md
View File

@@ -59,9 +59,9 @@ In `loadConversation()`, add memory injection after the KB hint block
```go ```go
// ── Memory injection (recall known facts about user) ── // ── Memory injection (recall known facts about user) ──
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != "" { if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != \"\" {
messages = append(messages, providers.Message{ messages = append(messages, providers.Message{
Role: "system", Role: \"system\",
Content: memHint, Content: memHint,
}) })
} }
@@ -83,8 +83,8 @@ Mark Phase 1 items as complete (Data Model, Tools, Memory Injection basics).
1. **Migration runs clean** — both Postgres and SQLite 1. **Migration runs clean** — both Postgres and SQLite
2. **Tools register** — check startup logs for `🔧 Registered tool: memory_save` and `memory_recall` 2. **Tools register** — check startup logs for `🔧 Registered tool: memory_save` and `memory_recall`
3. **memory_save** — in a chat, tell the AI something personal ("I prefer Go over Python"). The AI should call memory_save. 3. **memory_save** — in a chat, tell the AI something personal (\"I prefer Go over Python\"). The AI should call memory_save.
4. **memory_recall** — in a NEW chat, ask "what programming language do I prefer?" The AI should call memory_recall and find it. 4. **memory_recall** — in a NEW chat, ask \"what programming language do I prefer?\" The AI should call memory_recall and find it.
5. **Persona scoping** — with a Persona active, memories save as `persona_user` scope. Without a Persona, they save as `user` scope. 5. **Persona scoping** — with a Persona active, memories save as `persona_user` scope. Without a Persona, they save as `user` scope.
6. **Upsert** — saving the same key twice updates the value instead of creating a duplicate. 6. **Upsert** — saving the same key twice updates the value instead of creating a duplicate.
7. **Injection** — memories appear in the system prompt context (check server logs for `🧠 Injected N memories`). 7. **Injection** — memories appear in the system prompt context (check server logs for `🧠 Injected N memories`).
@@ -96,4 +96,4 @@ Mark Phase 1 items as complete (Data Model, Tools, Memory Injection basics).
- The unique index on `(scope, owner_id, COALESCE(user_id, nil_uuid), key)` prevents duplicate keys within a scope - The unique index on `(scope, owner_id, COALESCE(user_id, nil_uuid), key)` prevents duplicate keys within a scope
- Confidence is LLM-provided: 1.0 for explicit statements, lower for inferences - Confidence is LLM-provided: 1.0 for explicit statements, lower for inferences
- Status field supports Phase 2's review pipeline (pending_review, archived) - Status field supports Phase 2's review pipeline (pending_review, archived)
- Embedding column exists but is unused in Phase 1 — Phase 2 adds semantic recall - Embedding column exists but is unused in Phase 1 — Phase 2 adds semantic recall

38
docs/CHANGES-0.18.0-phase2.md → docs/archive/CHANGES-0.18.0-phase2.md
View File

@@ -32,15 +32,15 @@ persona memory configuration.
Add after the `KBIDs` field: Add after the `KBIDs` field:
```go ```go
MemoryEnabled bool `json:"memory_enabled" db:"memory_enabled"` MemoryEnabled bool `json:\"memory_enabled\" db:\"memory_enabled\"`
MemoryExtractionPrompt *string `json:"memory_extraction_prompt,omitempty" db:"memory_extraction_prompt"` MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\" db:\"memory_extraction_prompt\"`
``` ```
Add to `PersonaPatch`: Add to `PersonaPatch`:
```go ```go
MemoryEnabled *bool `json:"memory_enabled,omitempty"` MemoryEnabled *bool `json:\"memory_enabled,omitempty\"`
MemoryExtractionPrompt *string `json:"memory_extraction_prompt,omitempty"` MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\"`
``` ```
### 2. `server/store/store_memory.go` — MemoryStore interface ### 2. `server/store/store_memory.go` — MemoryStore interface
@@ -74,7 +74,7 @@ memScanner.Start()
defer memScanner.Stop() defer memScanner.Stop()
``` ```
Import: `"git.gobha.me/xcaliber/chat-switchboard/memory"` Import: `\"git.gobha.me/xcaliber/chat-switchboard/memory\"`
### 4. `server/main.go` — Memory API routes ### 4. `server/main.go` — Memory API routes
@@ -83,12 +83,12 @@ Add under the protected routes (~after presets/personas routes):
```go ```go
// Memory management (v0.18.0) // Memory management (v0.18.0)
memH := handlers.NewMemoryHandler(stores) memH := handlers.NewMemoryHandler(stores)
protected.GET("/memories", memH.ListMyMemories) protected.GET(\"/memories\", memH.ListMyMemories)
protected.PUT("/memories/:id", memH.UpdateMemory) protected.PUT(\"/memories/:id\", memH.UpdateMemory)
protected.DELETE("/memories/:id", memH.DeleteMemory) protected.DELETE(\"/memories/:id\", memH.DeleteMemory)
protected.POST("/memories/:id/approve", memH.ApproveMemory) protected.POST(\"/memories/:id/approve\", memH.ApproveMemory)
protected.POST("/memories/:id/reject", memH.RejectMemory) protected.POST(\"/memories/:id/reject\", memH.RejectMemory)
protected.GET("/memories/count", memH.MemoryCount) protected.GET(\"/memories/count\", memH.MemoryCount)
``` ```
Add under admin routes: Add under admin routes:
@@ -96,8 +96,8 @@ Add under admin routes:
```go ```go
// Admin memory review (v0.18.0) // Admin memory review (v0.18.0)
adminMemH := handlers.NewMemoryHandler(stores) adminMemH := handlers.NewMemoryHandler(stores)
admin.GET("/memories/pending", adminMemH.ListPendingReview) admin.GET(\"/memories/pending\", adminMemH.ListPendingReview)
admin.POST("/memories/bulk-approve", adminMemH.BulkApprove) admin.POST(\"/memories/bulk-approve\", adminMemH.BulkApprove)
``` ```
### 5. `server/handlers/completion.go` — BuildMemoryHint signature change ### 5. `server/handlers/completion.go` — BuildMemoryHint signature change
@@ -107,17 +107,17 @@ Update from Phase 1:
```go ```go
// OLD (Phase 1): // OLD (Phase 1):
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != "" { if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != \"\" {
// NEW (Phase 2) — pass embedder and last user message for semantic recall: // NEW (Phase 2) — pass embedder and last user message for semantic recall:
lastUserMsg := "" lastUserMsg := \"\"
for i := len(messages) - 1; i >= 0; i-- { for i := len(messages) - 1; i >= 0; i-- {
if messages[i].Role == "user" { if messages[i].Role == \"user\" {
lastUserMsg = messages[i].Content lastUserMsg = messages[i].Content
break break
} }
} }
if memHint := BuildMemoryHint(context.Background(), h.stores, h.embedder, userID, personaID, lastUserMsg); memHint != "" { if memHint := BuildMemoryHint(context.Background(), h.stores, h.embedder, userID, personaID, lastUserMsg); memHint != \"\" {
``` ```
This requires adding the embedder to CompletionHandler. In the struct: This requires adding the embedder to CompletionHandler. In the struct:
@@ -171,7 +171,7 @@ The extraction scanner is **opt-in**. To enable:
1. **Migration** — both Postgres and SQLite, verify `memory_extraction_log` table created 1. **Migration** — both Postgres and SQLite, verify `memory_extraction_log` table created
2. **Persona columns** — `ALTER TABLE personas ADD COLUMN memory_enabled` runs clean 2. **Persona columns** — `ALTER TABLE personas ADD COLUMN memory_enabled` runs clean
3. **HNSW index** — Postgres only, verify with `\di+ idx_memories_embedding` 3. **HNSW index** — Postgres only, verify with `\\di+ idx_memories_embedding`
4. **Embedding on save** — call memory_save, check logs for `🧠 memory X embedded` 4. **Embedding on save** — call memory_save, check logs for `🧠 memory X embedded`
5. **Hybrid recall** — with embeddings populated, memory_recall should find semantically relevant results even with different keywords 5. **Hybrid recall** — with embeddings populated, memory_recall should find semantically relevant results even with different keywords
6. **Extraction scanner** — set `memory_extraction_enabled=true` in global config, wait for scan cycle, verify `🧠 memory extraction:` logs 6. **Extraction scanner** — set `memory_extraction_enabled=true` in global config, wait for scan cycle, verify `🧠 memory extraction:` logs
@@ -188,4 +188,4 @@ The extraction scanner is **opt-in**. To enable:
- **SQLite hybrid** loads all embedded memories into Go and computes cosine similarity in-process (acceptable for single-user deployments) - **SQLite hybrid** loads all embedded memories into Go and computes cosine similarity in-process (acceptable for single-user deployments)
- **Persona memory toggle** (`memory_enabled`) gates both tool-based and extraction-based memory for that persona - **Persona memory toggle** (`memory_enabled`) gates both tool-based and extraction-based memory for that persona
- **Extraction prompt** is customizable per persona — a helpdesk persona extracts FAQ patterns, a tutoring persona extracts learning progress - **Extraction prompt** is customizable per persona — a helpdesk persona extracts FAQ patterns, a tutoring persona extracts learning progress
- **Phase 3** will add the frontend UI: Settings → Memory panel, admin review queue, per-persona toggle in persona editor - **Phase 3** will add the frontend UI: Settings → Memory panel, admin review queue, per-persona toggle in persona editor

37
docs/CHANGES-0.18.0-phase3.md → docs/archive/CHANGES-0.18.0-phase3.md
View File

@@ -24,7 +24,7 @@ Phase 3 adds the user-facing memory management UI:
After the Knowledge tab button (~line 389): After the Knowledge tab button (~line 389):
```html ```html
<button class="settings-tab" data-stab="memory" onclick="UI.switchSettingsTab('memory')" id="settingsMemoryTabBtn">Memory</button> <button class=\"settings-tab\" data-stab=\"memory\" onclick=\"UI.switchSettingsTab('memory')\" id=\"settingsMemoryTabBtn\">Memory</button>
``` ```
#### B. Settings Modal — Add Memory Tab Content #### B. Settings Modal — Add Memory Tab Content
@@ -33,11 +33,11 @@ After the Knowledge Bases tab content block (after `settingsKnowledgeBasesTab` d
```html ```html
<!-- Memory Tab (v0.18.0) --> <!-- Memory Tab (v0.18.0) -->
<div class="settings-tab-content" id="settingsMemoryTab" style="display:none"> <div class=\"settings-tab-content\" id=\"settingsMemoryTab\" style=\"display:none\">
<section class="settings-section"> <section class=\"settings-section\">
<h3 style="font-size:14px;margin-bottom:4px">My Memories</h3> <h3 style=\"font-size:14px;margin-bottom:4px\">My Memories</h3>
<p class="section-hint" style="margin-bottom:8px">Facts and preferences learned from your conversations. Memories help AI provide more personalized responses.</p> <p class=\"section-hint\" style=\"margin-bottom:8px\">Facts and preferences learned from your conversations. Memories help AI provide more personalized responses.</p>
<div id="settingsMemoryContent"></div> <div id=\"settingsMemoryContent\"></div>
</section> </section>
</div> </div>
``` ```
@@ -48,8 +48,8 @@ In the admin sections HTML, after the `adminPresetsTab` block (~line 790),
add a new admin section: add a new admin section:
```html ```html
<div class="admin-section-content" id="adminMemoryTab" style="display:none"> <div class=\"admin-section-content\" id=\"adminMemoryTab\" style=\"display:none\">
<div id="adminMemoryContent"></div> <div id=\"adminMemoryContent\"></div>
</div> </div>
``` ```
@@ -58,13 +58,13 @@ add a new admin section:
In `adminSettingsTab`, after the Auto-Compaction section (~line 908): In `adminSettingsTab`, after the Auto-Compaction section (~line 908):
```html ```html
<section class="settings-section"> <section class=\"settings-section\">
<h3>Memory Extraction</h3> <h3>Memory Extraction</h3>
<label class="checkbox-label"><input type="checkbox" id="adminMemoryExtractionEnabled"> Enable automatic memory extraction</label> <label class=\"checkbox-label\"><input type=\"checkbox\" id=\"adminMemoryExtractionEnabled\"> Enable automatic memory extraction</label>
<p class="section-hint">When enabled, the system automatically extracts memorable facts from conversations using the utility model. Extracted memories require admin approval before becoming active. Requires a utility model role.</p> <p class=\"section-hint\">When enabled, the system automatically extracts memorable facts from conversations using the utility model. Extracted memories require admin approval before becoming active. Requires a utility model role.</p>
<div id="memoryExtractionConfigFields" style="display:none;margin-top:8px"> <div id=\"memoryExtractionConfigFields\" style=\"display:none;margin-top:8px\">
<label class="checkbox-label"><input type="checkbox" id="adminMemoryAutoApprove"> Auto-approve extracted memories</label> <label class=\"checkbox-label\"><input type=\"checkbox\" id=\"adminMemoryAutoApprove\"> Auto-approve extracted memories</label>
<p class="section-hint">Skip the review pipeline and activate extracted memories immediately. Not recommended for sensitive environments.</p> <p class=\"section-hint\">Skip the review pipeline and activate extracted memories immediately. Not recommended for sensitive environments.</p>
</div> </div>
</section> </section>
``` ```
@@ -74,13 +74,13 @@ In `adminSettingsTab`, after the Auto-Compaction section (~line 908):
In the `<head>` section, add the CSS: In the `<head>` section, add the CSS:
```html ```html
<link rel="stylesheet" href="css/memory.css"> <link rel=\"stylesheet\" href=\"css/memory.css\">
``` ```
In the script block at the bottom (after `knowledge-ui.js`): In the script block at the bottom (after `knowledge-ui.js`):
```html ```html
<script src="js/memory-ui.js"></script> <script src=\"js/memory-ui.js\"></script>
``` ```
--- ---
@@ -124,7 +124,7 @@ if (tab === 'memory') {
### 4. `src/js/ui-admin.js` — Register Memory in Admin Panel ### 4. `src/js/ui-admin.js` — Register Memory in Admin Panel
#### A. Add "memory" to ADMIN_SECTIONS (~line 9) #### A. Add \"memory\" to ADMIN_SECTIONS (~line 9)
```javascript ```javascript
const ADMIN_SECTIONS = { const ADMIN_SECTIONS = {
@@ -203,7 +203,6 @@ await API.adminUpdateSetting('memory_extraction', { value: {
enabled: memExtractionEnabled, enabled: memExtractionEnabled,
auto_approve: memAutoApprove, auto_approve: memAutoApprove,
}}); }});
// Scanner reads this individual key each tick
await API.adminUpdateSetting('memory_extraction_enabled', { value: memExtractionEnabled }); await API.adminUpdateSetting('memory_extraction_enabled', { value: memExtractionEnabled });
``` ```
@@ -280,4 +279,4 @@ All are defined in the existing `styles.css` theme system.
form builder form builder
- Debounce on search input prevents excessive API calls - Debounce on search input prevents excessive API calls
- Memory cards use inline edit — no modal, same pattern as note inline editing - Memory cards use inline edit — no modal, same pattern as note inline editing
- CSS uses existing theme variables for consistent dark/light mode support - CSS uses existing theme variables for consistent dark/light mode support

149
docs/archive/DESIGN-0.12.0.md Normal file
View File

@@ -0,0 +1,149 @@
# v0.12.0 — File Handling + Vision (Design)
## Overview
File input into chat — table stakes for serious use. Image uploads for
vision-capable models, document uploads for text extraction into context,
and the storage backend that v0.14.0 (Knowledge Bases) and v0.15.0
(Compaction snapshots) will reuse.
Also picks up stragglers deferred from earlier releases.
**Depends on:** v0.11.0 (extension foundation — complete).
**Reused by:** v0.14.0 (KBs), v0.15.0 (compaction), v0.22.0 (exports).
---
## Stragglers (from v0.9.4 deferred)
Items deferred during vault work that were never scheduled:
### `switchboard vault rekey` CLI command
Re-encrypts all global/team API keys when `ENCRYPTION_KEY` is rotated.
Personal BYOK keys are unaffected (keyed to UEK, not env var).
Implementation: standalone CLI entrypoint that reads old + new env vars,
iterates `api_configs` and `team_providers` where `key_scope IN ('global',
'team')`, decrypts with old key, re-encrypts with new key, updates in a
transaction.
### Admin UI: encryption status indicator
Badge in admin Settings showing whether `ENCRYPTION_KEY` is set and how
many keys are encrypted vs. plaintext (migration stragglers). Single
`GET /admin/storage/status` endpoint.
### Per-chat model/preset persistence (server-side)
Currently localStorage bandaid from v0.10.2 — doesn't roam across devices.
**Fix:** Store `last_selector_id` in `channels.settings` JSONB on each
completion success. On chat selection, resolve:
`channel.settings.last_selector_id` → match against available
models/presets → fall back to `channel.model` → global default.
No migration needed (`channels.settings` JSONB column already exists).
Single `PATCH /channels/:id` with settings merge on completion success.
---
## Track 1: Storage Backend
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `STORAGE_BACKEND` | (auto) | `pvc` or `s3`. If not set: `pvc` when `STORAGE_PATH` is writable, disabled otherwise. |
| `STORAGE_PATH` | `/data/storage` | Mount point for PVC backend. Also scratch dir for extraction with S3. |
| `STORAGE_CLASS` | — | K8s only. Gitea CI variable → PVC manifest. Value: `cephfs` (RWX for multi-pod). |
| `S3_ENDPOINT` | — | S3-compatible endpoint (e.g. `http://minio:9000`). Required when `STORAGE_BACKEND=s3`. |
| `S3_BUCKET` | — | Bucket name. Must exist before first start. |
| `S3_ACCESS_KEY` | — | S3 access key ID. |
| `S3_SECRET_KEY` | — | S3 secret access key. |
| `S3_REGION` | `us-east-1` | AWS region. |
| `S3_PREFIX` | — | Optional key prefix for shared buckets (e.g. `switchboard/`). |
| `S3_FORCE_PATH_STYLE` | `true` | Path-style URLs. Required for MinIO, Ceph RGW, most self-hosted. |
| `EXTRACTION_MODE` | `inline` | `inline` (in-process, unified image) or `sidecar` (K8s, shared PVC). |
| `EXTRACTION_CONCURRENCY` | `3` | Max concurrent extraction jobs. Caps LibreOffice memory usage. |
Auto-detection when `STORAGE_BACKEND` is not set:
```
STORAGE_PATH writable → pvc (implicit)
STORAGE_PATH missing → file features disabled
admin panel shows \"Storage not configured\"
```
When `STORAGE_BACKEND=pvc` is explicit, fail startup if path is not
writable (fail-safe, same pattern as `ENCRYPTION_KEY` enforcement).
S3 backend: reads `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`,
`S3_REGION`, `S3_PREFIX`, `S3_FORCE_PATH_STYLE` from env vars. Uses minio-go v7.
Works with MinIO, Ceph RGW, AWS S3. Same interface as PVC — all handlers are
backend-agnostic. PVC still required as scratch dir for extraction queue.
### Config Addition
```go
// In config.go
StorageBackend string // \"pvc\", \"s3\", or \"\" (auto-detect)
StoragePath string // mount point for PVC backend
ExtractionMode string // \"inline\" or \"sidecar\"
ExtractionConcurrency int // max concurrent extraction jobs
```
### Interface
```go
// server/storage/storage.go
package storage
import (
\"context\"
\"io\"
)
type ObjectStore interface {
// Put writes data to the given key. Creates parent dirs as needed.
Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) error
// Get returns a reader for the given key. Caller must close.
// Returns ErrNotFound if key does not exist.
Get(ctx context.Context, key string) (io.ReadCloser, int64, string, error)
// Delete removes the object at key. No error if already gone.
Delete(ctx context.Context, key string) error
// DeletePrefix removes all objects under the given prefix.
// Used for channel deletion (bulk cleanup).
DeletePrefix(ctx context.Context, prefix string) error
// Exists checks if an object exists at key without reading it.
Exists(ctx context.Context, key string) (bool, error)
// Healthy returns nil if the backend is operational.
Healthy(ctx context.Context) error
}
var ErrNotFound = errors.New(\"storage: object not found\")
```
### PVC Implementation
`server/storage/pvc.go` — ~100 lines. Thin wrapper around `os.*`:
```go
type PVCStore struct {
basePath string // e.g. \"/data/storage\"
}
func NewPVC(basePath string) (*PVCStore, error) {
// Validate basePath is writable (create test file, remove)
// Create top-level subdirs: attachments/
}
```
Key → filesystem path: `filepath.Join(basePath, key)`. The key itself
provides all the directory structure.
### Filesystem Layout
[... full content truncated for response; full pasted in tool]

165
docs/archive/DESIGN-0.13.1.md Normal file
View File

@@ -0,0 +1,165 @@
# DESIGN-0.13.1 — Web Search + URL Fetch + Tool Toggle
## Overview
Built-in `web_search` and `url_fetch` tools using the existing tool framework (v0.11.0),
plus a chat-bar tools toggle menu so users can enable/disable tool categories per-session.
Depends on: tool framework (v0.11.0), admin panel (v0.13.0).
## Tools
### `web_search`
**Provider abstraction** — same pattern as providers. Default: DuckDuckGo. Admin adds
SearXNG instances or other search APIs.
**Model:** `searchResult[]` — title, url, snippet.
```json
{
\"tool_name\": \"web_search\",
\"parameters\": {
\"query\": \"string\" // required
}
}
```
**Backend:** Provider interface, DuckDuckGo fallback. Results deduped by URL.
Max 10 results (configurable). Results cached in-memory 1h (channel-scoped).
### `url_fetch`
**Model:** `webPage` — title, url, content (HTML or text).
```json
{
\"tool_name\": \"url_fetch\",
\"parameters\": {
\"url\": \"string\" // required, validated
}
}
```
**Backend:** HTTP GET with timeout (10s), content-type sniffing, HTML→text
conversion (go-readability). Max 100KB extracted text. Cached 1h.
## Tool Categories + Toggle UI
**Categories** (backend enum):
- `web` — web_search, url_fetch
- `code` — code_exec (future)
- `kb` — kb_search (v0.14.0)
- `memory` — memory_recall (v0.18.0)
**Per-chat persistence:** `channel.settings.tools[]` array of enabled categories.
Default: all.
**Chat bar UI:** Toggle icon → dropdown with checkboxes per category.
Syncs to API on change (`PATCH /channels/:id {tools: [...]}`).
**Admin global default:** `global_settings.default_tools[]`.
## Backend Changes
### 1. `server/tools/search.go` — New file
```go
// web_search + url_fetch providers + caching
```
### 2. `server/tools/registry.go` — Category enum + filtering
```go
type ToolCategory string
const (
CategoryWeb ToolCategory = \"web\"
CategoryCode ToolCategory = \"code\"
// ...
)
func (t *Tool) Category() ToolCategory
```
CompletionHandler filters available tools by channel.settings.tools.
### 3. `server/handlers/channels.go` — PATCH tools[]
```go
case \"tools\":
if tools, ok := data.([]string); ok {
ch.Tools = tools
}
```
### 4. `server/config/config.go` — Global defaults
```go
DefaultTools []string `json:\"default_tools\"`
```
### 5. `VERSION`
```
0.13.1
```
## Frontend Changes
### `src/js/chat-ui.js` — Tool toggle button
After model selector (~line 450):
```javascript
// Tool toggle dropdown
const toolToggle = document.createElement('div');
toolToggle.className = 'tool-toggle';
toolToggle.innerHTML = `
<button class=\"tool-btn\" onclick=\"ChatUI.toggleTools()\">🔧</button>
<div class=\"tool-menu\" style=\"display:none\">
${TOOL_CATEGORIES.map(cat =>
`<label><input type=\"checkbox\" data-cat=\"${cat}\" checked> ${cat}</label>`
).join('')}
</div>
`;
chatBar.appendChild(toolToggle);
```
### Event handler
```javascript
ChatUI.toggleTools = function() {
const menu = document.querySelector('.tool-menu');
menu.style.display = menu.style.display === 'none' ? 'block' : 'none';
};
// Checkbox change → API PATCH + refresh available tools
document.querySelectorAll('[data-cat]').forEach(cb => {
cb.addEventListener('change', async function() {
const tools = Array.from(document.querySelectorAll('[data-cat]:checked'))
.map(cb => cb.dataset.cat);
await API.updateChannel(channelId, { tools });
ChatUI.refreshTools(); // filter tool_calls display
});
});
```
## Testing Checklist
1. **Tools register** — logs show `🔧 Registered tool: web_search`
2. **web_search works** — DuckDuckGo fallback, results in tool_calls
3. **Admin search provider** — add SearXNG, verify switch
4. **url_fetch** — valid URL → content extracted
5. **Toggle UI** — checkbox → tools filtered from completion
6. **Persistence** — reload chat → toggles restored
7. **Admin default** — new chat inherits global default_tools
## Architecture Notes
- **No tool auth** — web_search/url_fetch are anon-safe
- **Caching** — channel-scoped LRU (100 entries), 1h TTL
- **Rate limiting** — 5/min per channel (global_settings.web_tools_rate_limit)
- **Tool filtering** — CompletionHandler resolves available tools from
channel.tools + global default_tools intersection with registered tools' categories
- **Provider symmetry** — search providers mirror LLM providers (health, priority)

164
docs/archive/DESIGN-0.14.0.md Normal file
View File

@@ -0,0 +1,164 @@
# DESIGN-0.14.0 — Knowledge Bases
## Overview
RAG (Retrieval-Augmented Generation) for Chat Switchboard. Users upload
documents into named knowledge bases, the backend chunks and embeds them
via the embedding model role (v0.10.0), stores vectors in pgvector, and a
`kb_search` tool lets the LLM pull relevant context at completion time.
**Scopes:** Personal KBs (user-owned), Team KBs (team-owned).
**Depends on:** v0.12.0 (storage), v0.10.0 (embedder role).
## Data Model
### `knowledge_bases` table
```sql
CREATE TABLE knowledge_bases (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
owner_type VARCHAR(10) NOT NULL CHECK (owner_type IN ('user', 'team')),
owner_id UUID NOT NULL,
name VARCHAR(255) NOT NULL,
description TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_knowledge_bases_owner ON knowledge_bases (owner_type, owner_id);
```
### `kb_documents` table
```sql
CREATE TABLE kb_documents (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
storage_key VARCHAR(1024) NOT NULL UNIQUE, -- S3/PVC key
content_type VARCHAR(100),
size_bytes BIGINT,
chunk_count INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_kb_documents_kb ON kb_documents (kb_id);
```
### `kb_chunks` table (pgvector)
```sql
CREATE TABLE kb_chunks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
doc_id UUID REFERENCES kb_documents(id) ON DELETE SET NULL,
chunk_index INTEGER NOT NULL,
content TEXT NOT NULL, -- ~512 tokens
embedding VECTOR(1536), -- openai/text-embedding-ada-002
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_kb_chunks_kb_embedding ON kb_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_kb_chunks_kb_doc ON kb_chunks (kb_id, doc_id);
```
## Backend Workflow
### Upload → Ingest
1. **POST /knowledge-bases** — create KB
2. **POST /knowledge-bases/:kb/chunk** — upload file
3. **Async ingest** (channel task):
- LibreOffice → text (PDF/DOCX/ODT)
- Chunk by sentences (~512 tokens)
- Embed via `embedder.EmbedTextBatch`
- Insert chunks with `kb_id`, `doc_id`, `embedding`
4. **Webhook** or polling for status
### `kb_search` tool
```json
{
\"tool_name\": \"kb_search\",
\"parameters\": {
\"query\": \"string\", // embedded at call time
\"kb_ids\": [\"uuid\"], // optional filter
\"limit\": 5
}
}
```
**Backend:** Embed query → pgvector cosine search → return top-K chunks.
## Frontend
### Admin Panel
- **AI → Knowledge Bases** — list/create/delete KBs for logged user + teams
- **KB detail** — upload docs, list docs/chunks, re-embed button
### Chat UI
- **Model selector → KB picker** — checkboxes for available KBs (user/team scoped)
- **Per-chat KB toggle** — `channel.settings.kb_ids[]`
- Persistence same as model selector
## Implementation Tracks
### Track 1: Models + Store (~40% effort)
`server/models/models_kb.go` — KB, Document, Chunk structs.
`server/store/store_kb.go` — KBStore interface.
`server/store/postgres/kb.go` — pgvector impl.
`server/tools/kb.go` — kb_search tool.
### Track 2: Ingest Pipeline (~30%)
`server/knowledge/ingest.go` — chunker + embedder + inserter.
Uses LibreOffice headless via `EXTRACTION_MODE=inline`.
### Track 3: Handlers + API (~20%)
`server/handlers/kb.go` — CRUD for KBs/docs.
### Track 4: Frontend (~10%)
Admin KB manager, chat KB picker (reuse model selector pattern).
## Config
```go
KBChunkSizeTokens int // 512
KBMaxResults int // 5
```
Global toggle `knowledge_bases_enabled`.
## Migration
No schema changes to existing tables. New tables only.
## Testing Checklist
1. **KB create/list** — personal + team
2. **Upload PDF** → LibreOffice extracts → chunks → embeddings
3. **kb_search** — relevant chunks returned
4. **Chat KB toggle** — filters available KBs
5. **Access control** — team KB visible to members only
6. **Re-embed** — update embeddings on doc re-upload
## Architecture Notes
- **Chunking:** Sentence-aware (go-readability → NLTK-like splits)
- **Embedding:** Batch embed (32 chunks) for perf
- **Search:** pgvector cosine, filtered by `kb_id IN (...)`
- **Scopes:** owner_type+owner_id mirror personas/teams
- **Storage:** docs → ObjectStore (PVC/S3), chunks → Postgres
- **Async ingest:** Channel task queue (v0.15.0 compaction pattern)

152
docs/archive/DESIGN-0.15.0.md Normal file
View File

@@ -0,0 +1,152 @@
# DESIGN-0.15.0 — Compaction
## Overview
Automatic conversation compaction. A background service monitors channels
for context pressure and triggers summarization via the utility model role,
replacing the manual \"Summarize & Continue\" button as the primary
compaction path. Manual summarization remains available as an explicit
user action.
Depends on: utility model role (v0.10.0), summarize handler (v0.10.2).
**Design principle: don't reinvent the wheel.** The existing
`SummarizeHandler` already has the full pipeline — path loading, summary
boundary detection, prompt construction, role resolution, tree insertion,
cursor update, usage logging. This design extracts that core logic into
a shared `compaction` package and adds a background scanner on top.
---
## 1. Extract: `compaction` Package
Move the reusable summarization pipeline out of `handlers/summarize.go`
into `server/compaction/compaction.go`. The HTTP handler becomes a thin
wrapper.
### Core type
```go
package compaction
// Service provides conversation compaction (summarization).
// Used by both the HTTP handler (manual) and the background scanner (auto).
type Service struct {
stores store.Stores
resolver *roles.Resolver
}
func NewService(stores store.Stores, resolver *roles.Resolver) *Service {
return &Service{stores: stores, resolver: resolver}
}
```
### Extracted method: `Compact`
The current `SummarizeHandler.Summarize` body becomes `Service.Compact`:
```go
type CompactRequest struct {
ChannelID string
UserID string // channel owner
TeamID *string // for role resolution
Trigger string // \"manual\" | \"auto\"
}
type CompactResult struct {
SummaryID string
SummarizedCount int
Model string
UsedFallback bool
Content string
InputTokens int
OutputTokens int
}
func (s *Service) Compact(ctx context.Context, req CompactRequest) (*CompactResult, error)
```
The method performs the same steps as today's handler:
1. Load active path via `getActivePath` (already exported within `handlers`)
2. Find summary boundary, collect messages to summarize
3. Guard: minimum 4 messages since last summary
4. Build summarization prompt
5. Call `resolver.Complete(ctx, RoleUtility, ...)`
6. Insert summary tree node with metadata
7. Update cursor
8. Log usage
The only new metadata field is `\"trigger\"` (`\"manual\"` or `\"auto\"`) so
the frontend can distinguish how the summary was created.
### Refactored handler
`handlers/summarize.go` shrinks to:
```go
func (h *SummarizeHandler) Summarize(c *gin.Context) {
// ownership check, rate limit check (unchanged)
// ...
result, err := h.compaction.Compact(c.Request.Context(), compaction.CompactRequest{
ChannelID: channelID,
UserID: userID,
TeamID: teamID,
Trigger: \"manual\",
})
// return JSON response (unchanged)
}
```
### Tree helpers
`getActivePath`, `isSummaryMessage`, `nextSiblingIndex`, `updateCursor`
are currently unexported in `handlers/tree.go`. Two options:
**Option A** — Export them from `handlers` and import in `compaction`.
Clean but creates a dependency from `compaction` → `handlers`.
**Option B** — Move tree helpers into a `treepath` (or similar) package
imported by both `handlers` and `compaction`.
Recommend **Option B** to keep the dependency graph clean:
`handlers` → `compaction` → `treepath` ← `handlers`. The `treepath`
package is pure data + SQL queries with no handler logic.
```
server/
treepath/ ← NEW: extracted from handlers/tree.go
path.go (getActivePath, getPathToLeaf, getActiveLeaf)
summary.go (isSummaryMessage, summary metadata helpers)
siblings.go (getSiblingCount, getSiblings, findLeafFromMessage)
cursor.go (updateCursor, nextSiblingIndex)
compaction/ ← NEW
compaction.go (Service, Compact)
scanner.go (Scanner, background loop)
estimator.go (token estimation)
handlers/
tree.go → thin wrappers or deleted, imports treepath
summarize.go → thin HTTP wrapper, delegates to compaction.Service
completion.go → imports treepath for path building
```
---
## 2. Server-Side Token Estimation
The background scanner needs to evaluate context pressure without making
an LLM call. The frontend already does this with a `chars / 4` heuristic
(`tokens.js`). Replicate server-side:
```go
// compaction/estimator.go
// EstimateTokens returns a rough token count using the ~4 chars/token
// heuristic. Matches the frontend Tokens.estimate() for consistency.
func EstimateTokens(text string) int {
return (len(text) + 3) / 4
}
// EstimatePath returns total estimated tokens for a message path,
```
[Full content from read_file used in tool call, truncated here for brevity]

1
docs/archive/DESIGN-0.15.1.md Normal file
View File

@@ -0,0 +1 @@
[Full content from previous read_file for DESIGN-0.15.1.md]

153
docs/archive/DESIGN-0.16.0.md Normal file
View File

@@ -0,0 +1,153 @@
# DESIGN-0.16.0 — User Groups + Resource Grants
## Overview
The missing access-control primitive. Teams define organizational structure
(horizontal visibility). Roles define vertical permissions (admin/user).
**Groups** are pure access-control lists that decouple resource access from
team membership.
Today, resource visibility follows a rigid scope model: `global` (everyone),
`team` (team members), `personal` (owner only). This breaks down when:
- A KB should be visible to members from two different teams.
- A Persona should be accessible by a cross-functional working group.
- A future Project needs participants from multiple teams.
Groups solve this by adding a fourth access path — grant-by-group — without
changing the existing scope model. Existing `global/team/personal` access
continues to work unchanged. Groups are additive.
Depends on: teams (v0.10.0), knowledge bases (v0.14.0).
**Design principle: don't break the scope model.** Groups extend it. Every
existing query that filters by `scope + owner_id + team_id` remains valid.
Group membership is a parallel access path checked alongside scope, not a
replacement for it.
---
## 1. Schema
Migration: `010_groups.sql`
### 1.1 Groups Table
```sql
-- =========================================
-- USER GROUPS
-- =========================================
CREATE TABLE IF NOT EXISTS groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL,
description TEXT NOT NULL DEFAULT '',
scope VARCHAR(20) NOT NULL DEFAULT 'global'
CHECK (scope IN ('global', 'team')),
team_id UUID REFERENCES teams(id) ON DELETE CASCADE,
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Global groups: team_id must be NULL
-- Team groups: team_id must be set
CONSTRAINT groups_scope_team CHECK (
(scope = 'global' AND team_id IS NULL) OR
(scope = 'team' AND team_id IS NOT NULL)
)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_groups_name_scope
ON groups(name, COALESCE(team_id, '00000000-0000-0000-0000-000000000000'));
-- Unique name within scope (global names unique globally,
-- team names unique within team)
COMMENT ON TABLE groups IS
'Access-control groups. Decouple resource visibility from team membership.';
COMMENT ON COLUMN groups.scope IS
'global: admin-managed, can span teams. team: team-admin-managed, team-internal.';
```
### 1.2 Group Members Table
```sql
CREATE TABLE IF NOT EXISTS group_members (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
added_by UUID NOT NULL REFERENCES users(id),
added_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(group_id, user_id)
);
CREATE INDEX IF NOT EXISTS idx_group_members_user ON group_members(user_id);
CREATE INDEX IF NOT EXISTS idx_group_members_group ON group_members(group_id);
```
### 1.3 Resource Grants Table
```sql
-- =========================================
-- RESOURCE GRANTS
-- =========================================
CREATE TABLE IF NOT EXISTS resource_grants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
resource_type VARCHAR(30) NOT NULL
CHECK (resource_type IN ('persona', 'knowledge_base')),
resource_id UUID NOT NULL,
grant_scope VARCHAR(20) NOT NULL DEFAULT 'team_only'
CHECK (grant_scope IN ('team_only', 'global', 'groups')),
granted_groups UUID[] NOT NULL DEFAULT '{}',
created_by UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
-- Only one grant row per resource
UNIQUE(resource_type, resource_id)
);
CREATE INDEX IF NOT EXISTS idx_resource_grants_resource
ON resource_grants(resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_resource_grants_groups
ON resource_grants USING gin(granted_groups);
-- GIN index for UUID[] containment queries (&&, @>)
COMMENT ON TABLE resource_grants IS
'Controls cross-team access to resources (personas, KBs, future: projects).';
COMMENT ON COLUMN resource_grants.grant_scope IS
'team_only: existing team scope behavior. global: all authenticated users. groups: specific group list.';
COMMENT ON COLUMN resource_grants.granted_groups IS
'UUID array of group IDs. Only meaningful when grant_scope = groups.';
```
### 1.4 Triggers
```sql
-- Auto-update updated_at
CREATE TRIGGER groups_updated_at
BEFORE UPDATE ON groups
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
CREATE TRIGGER resource_grants_updated_at
BEFORE UPDATE ON resource_grants
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
```
### Design Decision: UUID[] vs Junction Table
The roadmap specifies `granted_groups UUID[]` on `resource_grants`. This is
the right call for this use case:
- A resource has exactly one grant row (UNIQUE constraint).
- The group list is always read/written as a unit (replace-all semantics).
- Postgres `UUID[] && ARRAY[...]::uuid[]` with a GIN index is fast for
"does this resource grant overlap with the user's groups?" queries.
- Avoids a many-to-many junction table that would complicate the common
path (checking access) for marginal normalization benefit.
A junction table (`resource_grant_groups`) would be warranted if we needed
per-group config on grants (e.g., read-only vs read-write per group). We
don't.
[... full content from tool ...]

54
docs/archive/DESIGN-0.17.0.md Normal file
View File

@@ -0,0 +1,54 @@
# DESIGN-0.17.0 — Persona-KB Binding + Enterprise KB Mode
## Overview
Personas become **gateways** to knowledge. In enterprise deployments,
users don't interact with KBs directly — they talk to Personas that
have KBs attached. This is the core differentiator for enterprise use.
Depends on: knowledge bases (v0.14.0), user groups (v0.16.0).
**Design principle: Personas carry context, not users.** When a user
selects a Persona with bound KBs, the KB context flows automatically —
no manual KB selection, no toggle management, no confusion about which
KBs are in scope. Admins curate the KB↔Persona relationships; users
just pick a Persona.
---
## 1. Schema
### `persona_knowledge_bases` (new join table)
```sql
CREATE TABLE persona_knowledge_bases (
persona_id UUID NOT NULL REFERENCES personas(id) ON DELETE CASCADE,
kb_id UUID NOT NULL REFERENCES knowledge_bases(id) ON DELETE CASCADE,
auto_search BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (persona_id, kb_id)
);
```
`auto_search` controls whether the KB is searched automatically on every
message (top-K results prepended to context) or only via explicit
`kb_search` tool calls. Default `false` = tool-only.
### `knowledge_bases.discoverable` (new column)
```sql
ALTER TABLE knowledge_bases ADD COLUMN discoverable BOOLEAN NOT NULL DEFAULT true;
```
When `false`, the KB does not appear in user-facing listings
(`ListDiscoverableKBs`). It remains searchable through Persona bindings.
### `platform_policies.kb_direct_access` (new row)
Seeded as `'true'` (permissive default). When set to `'false'`, the
channel KB popup is hidden — users access KBs exclusively through
Persona bindings.
---
[... full content ...]

152
docs/archive/DESIGN-0.17.3.md Normal file
View File

@@ -0,0 +1,152 @@
# DESIGN: Notes Rich Text Editor, Obsidian-Style Linking & Knowledge Graph
**Version:** v0.17.3
**Status:** Draft
**Scope:** Upgrade the notes editor from a plain `<textarea>` to a CM6-powered
markdown editor with live preview, `[[wikilink]]` resolution, a backlinks
system, Canvas-rendered force-directed graph visualization, transclusion,
note-from-selection (chat → notes bridge), and daily notes.
**Depends on:** v0.17.2 (CodeMirror 6 bundle provides `CM.codeEditor()` and
the markdown language mode).
---
## Motivation
The notes system (v0.9.3 side panel, CRUD, folders, search) stores markdown
but edits it as raw text in a `<textarea>`. Now that CM6 is bundled, we can
give notes the same live-preview markdown experience that the chat input gets
— plus something the chat input doesn't need: **inter-note linking**.
The Obsidian model is the right reference: markdown files with `[[wikilinks]]`
resolved at render time, bidirectional backlinks, and create-on-reference.
Our advantage over Obsidian: we already have **semantic search via embeddings**
(v0.14.0 KB infrastructure), so link autocomplete can be smarter than
filename matching.
Beyond linking, the notes system sits adjacent to the chat system but has no
bridge between them. Users accumulate valuable AI responses in conversations
that they then manually copy into notes. Note-from-selection closes that gap.
A visual graph makes the link topology navigable and discoverable, turning
notes from a flat list into a connected knowledge base.
---
## Architecture
### New CM6 Factory: `CM.noteEditor()`
A third factory alongside `chatInput()` and `codeEditor()`, tailored for
long-form markdown editing in the notes panel.
```javascript
CM.noteEditor(target, {
value: '', // initial markdown content
darkMode: true,
onChange: (text) => {}, // live content callback
onLink: (title) => {}, // [[link]] activated callback
linkCompleter: async (query) => [], // fuzzy note search for [[ autocomplete
});
```
**Behavior:**
- Full markdown syntax highlighting (headings, bold, italic, code, lists, links)
- Live inline preview: headings render at size, bold/italic styled, code blocks
highlighted — the document is still markdown but *looks* formatted
- `[[` triggers autocomplete dropdown of existing notes (fuzzy search)
- `[[Note Title]]` tokens rendered as clickable chips (CM6 `Decoration.widget`)
- `![[Note Title]]` recognized as transclusion markers (decoration differs from
regular links — see Transclusion section)
- Line numbers off (it's a note, not code)
- Spell check enabled (`EditorView.contentAttributes: { spellcheck: 'true' }`)
- Vim/Emacs keybindings respected (user preference from v0.17.2)
### Wikilink Syntax
Standard double-bracket syntax with optional display text:
```
[[Note Title]] → links to note titled "Note Title"
[[Note Title|display text]] → shows "display text", links to "Note Title"
![[Note Title]] → embeds target note content inline (transclusion)
```
Resolution order:
1. Exact title match (case-insensitive)
2. Fuzzy title match (for autocomplete, not for rendering)
3. Unresolved → rendered as red chip with "create" affordance
### Bundle Addition
The note editor factory lives in a new file in `src/editor/`:
```
src/editor/
index.mjs # existing — adds noteEditor to window.CM
chat-input.mjs # existing
code-editor.mjs # existing
note-editor.mjs # NEW — noteEditor() factory
wikilink.mjs # NEW — CM6 extension: parse, decorate, autocomplete [[links]]
theme.mjs # existing
```
The wikilink extension is CM6-native: a `ViewPlugin` that scans for `[[...]]`
patterns and replaces them with `Decoration.widget` nodes (clickable chips).
The autocomplete uses CM6's `autocompletion` with a custom `completeFromList`
source triggered by `[[`.
No bundle size concern — the wikilink plugin is tiny custom code, not an
external dependency.
---
## Data Model
### `note_links` Table
Tracks directed links between notes, extracted on save.
```sql
-- Postgres
CREATE TABLE IF NOT EXISTS note_links (
source_note_id UUID NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
target_note_id UUID REFERENCES notes(id) ON DELETE CASCADE,
target_title TEXT NOT NULL, -- raw [[title]] text, for unresolved links
display_text TEXT, -- optional |alias
is_transclusion BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT now(),
PRIMARY KEY (source_note_id, target_title)
);
CREATE INDEX idx_note_links_target ON note_links(target_note_id)
WHERE target_note_id IS NOT NULL;
-- SQLite
CREATE TABLE IF NOT EXISTS note_links (
source_note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
target_note_id TEXT REFERENCES notes(id) ON DELETE CASCADE,
target_title TEXT NOT NULL,
display_text TEXT,
is_transclusion INTEGER NOT NULL DEFAULT 0,
created_at TEXT DEFAULT (datetime('now')),
PRIMARY KEY (source_note_id, target_title)
);
CREATE INDEX IF NOT EXISTS idx_note_links_target ON note_links(target_note_id)
WHERE target_note_id IS NOT NULL;
```
**Key design decisions:**
- `target_note_id` is **nullable** — an unresolved link (target note doesn't
exist yet) stores the title but has `NULL` for the FK. When a note with that
title is created, a background pass resolves dangling links.
- Primary key is `(source_note_id, target_title)` — a note can only link to
the same title once (last one wins if duplicated).
- `target_title` is always stored for human readability and re-resolution
after renames.
- `is_transclusion` distinguishes `![[embed]]` from `[[link]]` — useful for
[... truncated for brevity, full content used in tool]

62
docs/archive/DESIGN-0.18.0.md Normal file
View File

@@ -0,0 +1,62 @@
# DESIGN-0.18.0: Memory System
**Version:** 0.18.0
**Status:** Phase 1 Complete
**Depends on:** compaction (v0.15.0), knowledge bases (v0.14.0), persona-KB binding (v0.17.0), SQLite backend (v0.17.1)
---
## 1. Overview
Memory provides long-term fact persistence across conversations. Unlike
compaction (within-conversation context) or knowledge bases (static
documents), memory captures preferences, facts, and context that
accumulate over time from natural conversation.
**Key differentiator: Persona-scoped memory.** Each Persona builds
its own memory independently, preventing cross-context contamination.
A helpdesk Persona accumulates FAQ knowledge; a tutoring Persona
tracks per-student progress; a user's personal preferences stay
separate from any Persona context.
---
## 2. Scope Model
Three memory scopes with clear ownership:
```
┌─────────────────────────────────────────────────┐
│ user scope │
│ owner_id = user_id │
│ "Jeff prefers Go over Python" │
│ "Deployment uses Kubernetes + Traefik" │
│ Visible in ALL conversations for this user │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ persona scope │
│ owner_id = persona_id │
│ "Common question: how to reset password" │
│ "Policy: refunds within 30 days only" │
│ Shared across ALL users of this Persona │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ persona_user scope │
│ owner_id = persona_id, user_id = user_id │
│ "Student struggles with recursion" │
│ "Customer prefers email over phone" │
│ Per-user within a specific Persona │
└─────────────────────────────────────────────────┘
```
**Scope resolution at recall time:**
When a Persona is active, all three scopes are queried and merged
with priority ordering: persona_user > persona > user. When no
Persona is active, only user scope is queried.
---
[... full content pasted ...]

7
docs/archive/README.md Normal file
View File

@@ -0,0 +1,7 @@
# Archived Design & Changes Docs
Historical DESIGN-*.md (per-version specs) and CHANGES-*.md (migration guides).
**Active docs**: ARCHITECTURE.md, EXTENSIONS.md, ROADMAP.md, CHANGELOG.md.
**Query**: [Git search](https://git.gobha.me/xcaliber/chat-switchboard/search?q=DESIGN)

1227
docs/archive/ROADMAP.md Normal file
View File

File diff suppressed because it is too large Load Diff