# DESIGN-0.14.0 — Knowledge Bases

## Overview

RAG (Retrieval-Augmented Generation) for Chat Switchboard. Users upload
documents into named knowledge bases, the backend chunks and embeds them
via the embedding model role (v0.10.0), stores vectors in pgvector, and a
`kb_search` tool lets the LLM pull relevant context at completion time.

**Scopes:** Personal KBs (user-owned), Team KBs (team-owned).

**Depends on:** v0.12.0 (storage), v0.10.0 (embedder role).

## Data Model

### `knowledge_bases` table

```sql
CREATE TABLE knowledge_bases (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    owner_type VARCHAR(10) NOT NULL CHECK (owner_type IN ('user', 'team')),
    owner_id UUID NOT NULL,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_knowledge_bases_owner ON knowledge_bases (owner_type, owner_id);
```

### `kb_documents` table

```sql
CREATE TABLE kb_documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    storage_key VARCHAR(1024) NOT NULL UNIQUE,  -- S3/PVC key
    content_type VARCHAR(100),
    size_bytes BIGINT,
    chunk_count INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_kb_documents_kb ON kb_documents (kb_id);
```

### `kb_chunks` table (pgvector)

```sql
CREATE TABLE kb_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
    doc_id UUID REFERENCES kb_documents(id) ON DELETE SET NULL,
    chunk_index INTEGER NOT NULL,
    content TEXT NOT NULL,  -- ~512 tokens
    embedding VECTOR(1536),  -- openai/text-embedding-ada-002
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_kb_chunks_kb_embedding ON kb_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_kb_chunks_kb_doc ON kb_chunks (kb_id, doc_id);
```

## Backend Workflow

### Upload → Ingest

1. **POST /knowledge-bases** — create KB
2. **POST /knowledge-bases/:kb/chunk** — upload file
3. **Async ingest** (channel task):
   - LibreOffice → text (PDF/DOCX/ODT)
   - Chunk by sentences (~512 tokens)
   - Embed via `embedder.EmbedTextBatch`
   - Insert chunks with `kb_id`, `doc_id`, `embedding`
4. **Webhook** or polling for status

### `kb_search` tool

```json
{
  \"tool_name\": \"kb_search\",
  \"parameters\": {
    \"query\": \"string\",  // embedded at call time
    \"kb_ids\": [\"uuid\"],  // optional filter
    \"limit\": 5
  }
}
```

**Backend:** Embed query → pgvector cosine search → return top-K chunks.

## Frontend

### Admin Panel

- **AI → Knowledge Bases** — list/create/delete KBs for logged user + teams
- **KB detail** — upload docs, list docs/chunks, re-embed button

### Chat UI

- **Model selector → KB picker** — checkboxes for available KBs (user/team scoped)
- **Per-chat KB toggle** — `channel.settings.kb_ids[]`
- Persistence same as model selector

## Implementation Tracks

### Track 1: Models + Store (~40% effort)

`server/models/models_kb.go` — KB, Document, Chunk structs.

`server/store/store_kb.go` — KBStore interface.

`server/store/postgres/kb.go` — pgvector impl.

`server/tools/kb.go` — kb_search tool.

### Track 2: Ingest Pipeline (~30%)

`server/knowledge/ingest.go` — chunker + embedder + inserter.

Uses LibreOffice headless via `EXTRACTION_MODE=inline`.

### Track 3: Handlers + API (~20%)

`server/handlers/kb.go` — CRUD for KBs/docs.

### Track 4: Frontend (~10%)

Admin KB manager, chat KB picker (reuse model selector pattern).

## Config

```go
KBChunkSizeTokens  int  // 512
KBMaxResults       int  // 5
```

Global toggle `knowledge_bases_enabled`.

## Migration

No schema changes to existing tables. New tables only.

## Testing Checklist

1. **KB create/list** — personal + team
2. **Upload PDF** → LibreOffice extracts → chunks → embeddings
3. **kb_search** — relevant chunks returned
4. **Chat KB toggle** — filters available KBs
5. **Access control** — team KB visible to members only
6. **Re-embed** — update embeddings on doc re-upload

## Architecture Notes

- **Chunking:** Sentence-aware (go-readability → NLTK-like splits)
- **Embedding:** Batch embed (32 chunks) for perf
- **Search:** pgvector cosine, filtered by `kb_id IN (...)`
- **Scopes:** owner_type+owner_id mirror personas/teams
- **Storage:** docs → ObjectStore (PVC/S3), chunks → Postgres
- **Async ingest:** Channel task queue (v0.15.0 compaction pattern)