164 lines
4.6 KiB
Markdown
164 lines
4.6 KiB
Markdown
# DESIGN-0.14.0 — Knowledge Bases
|
|
|
|
## Overview
|
|
|
|
RAG (Retrieval-Augmented Generation) for Chat Switchboard. Users upload
|
|
documents into named knowledge bases, the backend chunks and embeds them
|
|
via the embedding model role (v0.10.0), stores vectors in pgvector, and a
|
|
`kb_search` tool lets the LLM pull relevant context at completion time.
|
|
|
|
**Scopes:** Personal KBs (user-owned), Team KBs (team-owned).
|
|
|
|
**Depends on:** v0.12.0 (storage), v0.10.0 (embedder role).
|
|
|
|
## Data Model
|
|
|
|
### `knowledge_bases` table
|
|
|
|
```sql
|
|
CREATE TABLE knowledge_bases (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
owner_type VARCHAR(10) NOT NULL CHECK (owner_type IN ('user', 'team')),
|
|
owner_id UUID NOT NULL,
|
|
name VARCHAR(255) NOT NULL,
|
|
description TEXT,
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
|
);
|
|
|
|
CREATE INDEX idx_knowledge_bases_owner ON knowledge_bases (owner_type, owner_id);
|
|
```
|
|
|
|
### `kb_documents` table
|
|
|
|
```sql
|
|
CREATE TABLE kb_documents (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
|
|
name VARCHAR(255) NOT NULL,
|
|
storage_key VARCHAR(1024) NOT NULL UNIQUE, -- S3/PVC key
|
|
content_type VARCHAR(100),
|
|
size_bytes BIGINT,
|
|
chunk_count INTEGER DEFAULT 0,
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
|
);
|
|
|
|
CREATE INDEX idx_kb_documents_kb ON kb_documents (kb_id);
|
|
```
|
|
|
|
### `kb_chunks` table (pgvector)
|
|
|
|
```sql
|
|
CREATE TABLE kb_chunks (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
kb_id UUID REFERENCES knowledge_bases(id) ON DELETE CASCADE,
|
|
doc_id UUID REFERENCES kb_documents(id) ON DELETE SET NULL,
|
|
chunk_index INTEGER NOT NULL,
|
|
content TEXT NOT NULL, -- ~512 tokens
|
|
embedding VECTOR(1536), -- openai/text-embedding-ada-002
|
|
metadata JSONB DEFAULT '{}',
|
|
created_at TIMESTAMP DEFAULT NOW()
|
|
);
|
|
|
|
CREATE INDEX idx_kb_chunks_kb_embedding ON kb_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
|
|
CREATE INDEX idx_kb_chunks_kb_doc ON kb_chunks (kb_id, doc_id);
|
|
```
|
|
|
|
## Backend Workflow
|
|
|
|
### Upload → Ingest
|
|
|
|
1. **POST /knowledge-bases** — create KB
|
|
2. **POST /knowledge-bases/:kb/chunk** — upload file
|
|
3. **Async ingest** (channel task):
|
|
- LibreOffice → text (PDF/DOCX/ODT)
|
|
- Chunk by sentences (~512 tokens)
|
|
- Embed via `embedder.EmbedTextBatch`
|
|
- Insert chunks with `kb_id`, `doc_id`, `embedding`
|
|
4. **Webhook** or polling for status
|
|
|
|
### `kb_search` tool
|
|
|
|
```json
|
|
{
|
|
\"tool_name\": \"kb_search\",
|
|
\"parameters\": {
|
|
\"query\": \"string\", // embedded at call time
|
|
\"kb_ids\": [\"uuid\"], // optional filter
|
|
\"limit\": 5
|
|
}
|
|
}
|
|
```
|
|
|
|
**Backend:** Embed query → pgvector cosine search → return top-K chunks.
|
|
|
|
## Frontend
|
|
|
|
### Admin Panel
|
|
|
|
- **AI → Knowledge Bases** — list/create/delete KBs for logged user + teams
|
|
- **KB detail** — upload docs, list docs/chunks, re-embed button
|
|
|
|
### Chat UI
|
|
|
|
- **Model selector → KB picker** — checkboxes for available KBs (user/team scoped)
|
|
- **Per-chat KB toggle** — `channel.settings.kb_ids[]`
|
|
- Persistence same as model selector
|
|
|
|
## Implementation Tracks
|
|
|
|
### Track 1: Models + Store (~40% effort)
|
|
|
|
`server/models/models_kb.go` — KB, Document, Chunk structs.
|
|
|
|
`server/store/store_kb.go` — KBStore interface.
|
|
|
|
`server/store/postgres/kb.go` — pgvector impl.
|
|
|
|
`server/tools/kb.go` — kb_search tool.
|
|
|
|
### Track 2: Ingest Pipeline (~30%)
|
|
|
|
`server/knowledge/ingest.go` — chunker + embedder + inserter.
|
|
|
|
Uses LibreOffice headless via `EXTRACTION_MODE=inline`.
|
|
|
|
### Track 3: Handlers + API (~20%)
|
|
|
|
`server/handlers/kb.go` — CRUD for KBs/docs.
|
|
|
|
### Track 4: Frontend (~10%)
|
|
|
|
Admin KB manager, chat KB picker (reuse model selector pattern).
|
|
|
|
## Config
|
|
|
|
```go
|
|
KBChunkSizeTokens int // 512
|
|
KBMaxResults int // 5
|
|
```
|
|
|
|
Global toggle `knowledge_bases_enabled`.
|
|
|
|
## Migration
|
|
|
|
No schema changes to existing tables. New tables only.
|
|
|
|
## Testing Checklist
|
|
|
|
1. **KB create/list** — personal + team
|
|
2. **Upload PDF** → LibreOffice extracts → chunks → embeddings
|
|
3. **kb_search** — relevant chunks returned
|
|
4. **Chat KB toggle** — filters available KBs
|
|
5. **Access control** — team KB visible to members only
|
|
6. **Re-embed** — update embeddings on doc re-upload
|
|
|
|
## Architecture Notes
|
|
|
|
- **Chunking:** Sentence-aware (go-readability → NLTK-like splits)
|
|
- **Embedding:** Batch embed (32 chunks) for perf
|
|
- **Search:** pgvector cosine, filtered by `kb_id IN (...)`
|
|
- **Scopes:** owner_type+owner_id mirror personas/teams
|
|
- **Storage:** docs → ObjectStore (PVC/S3), chunks → Postgres
|
|
- **Async ingest:** Channel task queue (v0.15.0 compaction pattern) |