Changeset 0.14.0 (#67)

ROADMAP.md

@@ -50,21 +50,30 @@ v0.15.0 Compaction (utility role + background job)
           │
 v0.15.1 Context Recall Tools (attachment_recall, conversation_search)
           │
-v0.16.0 Conversation Memory (cross-conversation user facts)
+  ┌───────┴──────────────┐
+  │                      │
+v0.16.0 User Groups    v0.17.0 Persona-KB Binding
+  + Resource Grants      + Enterprise KB Mode
+  │                      │
+  └───────┬──────────────┘
           │
-v0.17.0 @mention Routing + Multi-model
+v0.18.0 Memory (user + persona scopes, review pipeline)
           │
-v0.18.0 Extension Surfaces + Modes
+v0.19.0 Projects / Workspaces (organizational containers)
           │
-v0.19.0 Smart Routing (policy on model roles)
+v0.20.0 @mention Routing + Multi-model
           │
-v0.20.0 Multi-Participant Channels + Presence
+v0.21.0 Extension Surfaces + Modes
           │
-v0.21.0 Auth Strategy (mTLS/OIDC) + Full RBAC
+v0.22.0 Smart Routing (policy on model roles)
+          │
+v0.23.0 Multi-Participant Channels + Presence
+          │
+v0.24.0 Auth Strategy (mTLS/OIDC) + Full RBAC + Group permissions
           │
   ┌───────┴──────────────┐
   │                      │
-v0.22.0 Workflow       v0.23.0 Tasks /
+v0.25.0 Workflow       v0.26.0 Tasks /
   Engine                 Autonomous Agents
   (team-owned,           (service channels,
    staged processes,      scheduler, unattended
@@ -263,7 +272,7 @@ after backfill. If `ENCRYPTION_KEY` is not set, migration refuses to run
 derivation. No additional UX. On login, derive PDK → unwrap UEK → cache
 in session. User never knows the vault exists.
 
-**mTLS / OIDC** (v0.21.0) — authentication is external (cert/token).
+**mTLS / OIDC** (v0.24.0) — authentication is external (cert/token).
 Vault passphrase is conditionally required:
 
 ```
@@ -345,7 +354,7 @@ func (s *Store) DecryptAPIKey(config ApiConfig, uekCache *sync.Map) (string, err
 - [x] UEK eviction on logout (memory zeroing)
 - [ ] UEK re-wrap on password change _(deferred — Settings handler)_
 - [ ] UEK destruction on admin password reset with confirmation _(deferred)_
-- [ ] Vault passphrase prompt for mTLS/OIDC _(deferred — v0.21.0 dependency)_
+- [ ] Vault passphrase prompt for mTLS/OIDC _(deferred — v0.24.0 dependency)_
 
 **Provider key lifecycle**
 - [x] Encrypt on provider create/update (all 6 write paths: admin global, personal, team)
@@ -688,16 +697,55 @@ Depends on: admin panel refactor (v0.13.0) for config UI, tool framework (v0.11.
 
 ## v0.14.0 — Knowledge Bases
 
-Depends on: embedding model role (v0.10.0), file storage (v0.12.0).
+RAG for Switchboard. Upload documents into named knowledge bases, chunk
+and embed via the embedding model role, store vectors in pgvector, and
+a `kb_search` tool lets the LLM pull relevant context at completion time.
 
-- [ ] Embedding pipeline: chunking (recursive, semantic), generation via embedding role
-- [ ] `pgvector` storage and similarity search
-- [ ] `knowledge_bases` table with team scoping
-- [ ] KB document storage via v0.12.0 storage backend
-- [ ] `kb_search` tool (uses existing tool framework)
-- [ ] Context injection in completion flow
-- [ ] Notes embedded too (once pipeline exists)
-- [ ] Per-channel KB toggle
+Depends on: embedding model role (v0.10.0), file storage (v0.12.0),
+tool framework (v0.11.0), admin panel (v0.13.0).
+See [DESIGN-0.14.0.md](DESIGN-0.14.0.md) for full spec.
+
+**Pre-Req: Embedding Dropdown Fix**
+- [ ] Tolerant type filter in `renderRoleConfig()` (include untyped models for embedding role)
+- [ ] Manual model ID entry fallback (pencil toggle: dropdown ↔ text input)
+- [ ] Auto-switch to manual entry when dropdown has zero options
+
+**Phase 1: Schema + Chunker**
+- [x] Migration `008_knowledge_bases.sql` (pgvector extension + tables)
+- [x] `pgvector-go` dependency in `go.mod`
+- [x] `knowledge/chunker.go` — recursive text splitter (configurable size/overlap/separators)
+- [x] `knowledge/chunker_test.go`
+- [x] `KnowledgeBase`, `KBDocument`, `KBChunk`, `KBSearchResult`, `ChannelKB` model types
+- [x] `KnowledgeBaseStore` interface + postgres implementation (CRUD, no vector ops yet)
+- [x] Wire `Stores.KnowledgeBases` in `main.go`
+
+**Phase 2: Ingestion Pipeline**
+- [x] `knowledge/embedder.go` — batch embed via role resolver (batches of 100)
+- [x] `knowledge/ingest.go` — orchestrator (extract → chunk → embed → store)
+- [x] Dimension normalization (zero-pad to 3072 column width)
+- [x] `handlers/knowledge_bases.go` — KB CRUD + document upload endpoints
+- [x] Async ingestion goroutine with semaphore(3) concurrency limit
+- [x] Document status polling endpoint
+- [x] `SimilaritySearch()` in store (pgvector cosine query)
+- [x] Admin panel: AI → Knowledge Bases section
+- [x] IVFFlat index creation
+
+**Phase 3: kb_search Tool + Channel Integration**
+- [x] `tools/kbsearch.go` — `kb_search` tool (closure injection, late registration)
+- [x] `channel_knowledge_bases` — per-channel KB toggle
+- [x] System prompt KB hint injection in completion handler
+- [x] KB toggle popup in chat input bar
+- [x] "knowledge" category in tools toggle menu
+- [x] Team KB endpoints + team management UI tab
+- [x] Personal KB endpoints + settings UI
+
+**Phase 4: Notes + Polish**
+- [x] Notes `embedding` column (`vector(3072)`)
+- [x] Semantic note search (`note_search` tool `semantic: true` param)
+- [x] KB rebuild endpoint (re-chunk + re-embed all docs)
+- [x] Audit logging for KB operations
+- [x] Usage tracking for embedding calls (`role = 'embedding'`)
+- [x] Integration tests
 
 ---
 
@@ -741,31 +789,187 @@ may need to re-read source material. These tools bridge that gap.
 - [ ] Documents: count extracted text tokens
 - [ ] Staged attachments reflected in input token counter before send
 
+**KB auto-injection** (depends on: knowledge bases v0.14.0, compaction v0.15.0)
+- [ ] Pre-pend top-K KB results to system prompt (automatic, no tool call needed)
+- [ ] Context budget aware: skip injection if context already near limit
+- [ ] Per-channel toggle: tool-only vs auto-inject vs both
+
 ---
 
-## v0.16.0 — Conversation Memory
+## v0.16.0 — User Groups + Resource Grants
 
-Depends on: compaction (v0.15.0), knowledge bases (v0.14.0).
+The missing access-control primitive. Teams define organizational
+structure (horizontal visibility). Roles define vertical permissions
+(admin/user). **Groups** are pure access-control lists that decouple
+resource access from team membership.
 
-Long-term memory across conversations. Unlike KB (static documents) or
-compaction (within-conversation), this captures user-level facts and
-preferences that persist across channels.
+Depends on: teams (v0.10.0), knowledge bases (v0.14.0).
 
-- [ ] `user_memory` table: `user_id`, `key`, `value`, `source_channel_id`, `confidence`, `created_at`, `updated_at`
-- [ ] `memory_save` tool: LLM can explicitly store a fact ("user prefers Python", "project deadline is March 15")
+**Groups Entity**
+- [ ] `groups` table: `id`, `name`, `description`, `scope` (global, team), `team_id` (nullable — team-scoped groups), `created_by`, `created_at`, `updated_at`
+- [ ] `group_members` table: `group_id`, `user_id`, `added_by`, `added_at`
+- [ ] A user can belong to 0 or more groups
+- [ ] Groups can span multiple teams (global scope) or be team-internal
+- [ ] Admin CRUD: create/delete groups, add/remove members
+- [ ] Team admin can manage team-scoped groups
+
+**Resource Grant Model**
+- [ ] `resource_grants` table: `resource_type` (persona, knowledge_base, project), `resource_id`, `grant_scope` (team_only, global, groups), `granted_groups` (UUID[]), `created_by`
+- [ ] Extends existing `persona_grants` pattern to all resource types
+- [ ] Three-way access: `team_only` (existing team members), `global` (all authenticated), `groups` (specific group list)
+- [ ] Grant resolution: check user's group memberships against resource grants
+- [ ] Backward compatible: existing team/global scoping continues to work, groups are additive
+
+**API Endpoints**
+- [ ] CRUD: `/api/v1/admin/groups`, `/api/v1/teams/:teamId/groups`
+- [ ] Membership: `POST/DELETE /api/v1/groups/:id/members`
+- [ ] Grant management on resources: `PUT /api/v1/knowledge-bases/:id/grants`, `PUT /api/v1/presets/:id/grants`
+- [ ] User-facing: `GET /api/v1/groups/mine` (groups I belong to)
+
+**Frontend**
+- [ ] Admin panel: Groups section (CRUD, member management)
+- [ ] Team admin: group management tab
+- [ ] Resource grant picker: when creating/editing a Persona or KB, choose Team Only / Global / Select Groups
+
+---
+
+## v0.17.0 — Persona-KB Binding + Enterprise KB Mode
+
+Personas become **gateways** to knowledge. In enterprise deployments,
+users don't interact with KBs directly — they talk to Personas that
+have KBs attached. This is the core differentiator for enterprise use.
+
+Depends on: knowledge bases (v0.14.0), user groups (v0.16.0).
+
+**Persona-KB Binding**
+- [ ] `persona_knowledge_bases` table: `persona_id`, `kb_id`, `auto_search` (bool — always search vs tool-only)
+- [ ] When a channel uses a Persona with bound KBs, `kb_search` is auto-scoped to those KBs
+- [ ] No user toggle needed — Persona carries its own KB context
+- [ ] Channel KB toggle becomes "additional KBs" on top of Persona-provided ones
+- [ ] Persona system prompt auto-includes KB listing hint (reuses `BuildKBHint` pattern)
+- [ ] Admin/team admin UI: KB picker on Persona create/edit form
+
+**Enterprise KB Mode**
+- [ ] `discoverable` flag on knowledge_bases: `true` (default — appears in user's KB popup) or `false` (hidden, only accessible through Persona binding)
+- [ ] Hidden KBs don't appear in `GET /api/v1/knowledge-bases` for non-admins
+- [ ] Hidden KBs still searchable by `kb_search` tool when accessed through a Persona
+- [ ] Admin panel: toggle discoverability per KB
+- [ ] Platform policy: `kb_direct_access` — when `false`, disables channel KB popup entirely (strict enterprise mode)
+
+**Access Control Integration**
+- [ ] Persona grants (v0.16.0 groups) control who can *use* a Persona
+- [ ] KB grants control who can *manage* a KB (upload/delete docs)
+- [ ] Persona-KB binding grants implicit *search* access through the Persona
+- [ ] Users who can use a Persona can search its KBs, even if they can't see those KBs directly
+
+**Team Admin Workflow**
+- [ ] Team admin creates KB → uploads documents → marks non-discoverable
+- [ ] Team admin creates Persona → binds KBs → sets access to Team Only or specific Groups
+- [ ] Team members see the Persona in their preset list, use it, get KB-powered answers
+- [ ] Team members never see or manage the underlying KBs
+
+---
+
+## v0.18.0 — Memory (User + Persona Scopes)
+
+Long-term memory across conversations, with scope-aware isolation.
+Unlike KB (static documents) or compaction (within-conversation), this
+captures facts and preferences that persist across channels.
+
+**Key differentiator: Persona-scoped memory.** Each Persona can build
+its own memory independently of user memory, preventing cross-context
+contamination and enabling specialized knowledge accumulation.
+
+Depends on: compaction (v0.15.0), knowledge bases (v0.14.0), persona-KB binding (v0.17.0).
+
+**Memory Scopes**
+- [ ] `user` — personal facts/preferences, persists across all conversations (current industry standard)
+- [ ] `persona` — shared across all users of that Persona, accumulated from conversations (differentiator)
+- [ ] `persona+user` — per-user within a Persona context (e.g. tutoring progress per student)
+- [ ] Configurable per Persona: which scopes are active, whether user memory is passed through
+
+**Data Model**
+- [ ] `memories` table: `id`, `scope` (user, persona, persona_user), `owner_id` (user_id or persona_id), `user_id` (nullable — for persona+user scope), `key`, `value`, `source_channel_id`, `confidence` (float), `status` (active, pending_review, archived), `created_at`, `updated_at`
+- [ ] Index on `(scope, owner_id, user_id)` for fast lookup
+- [ ] Embedding column (`vector(3072)`) for semantic memory search
+
+**Tools**
+- [ ] `memory_save` tool: LLM explicitly stores a fact
+    - Scope-aware: saves to the active memory scope for the current Persona context
+    - Structured: `key` (short label) + `value` (detail) + `confidence`
 - [ ] `memory_recall` tool: LLM queries stored facts relevant to current context
-- [ ] Auto-extraction: background job identifies memorable facts from conversations (opt-in)
-- [ ] Memory injection: relevant facts injected into system prompt at completion time
-- [ ] User controls: view, edit, delete memories in Settings
-- [ ] Admin controls: enable/disable, retention policies, per-team scoping
-- [ ] Privacy: memories never cross team boundaries
+    - Merges results from applicable scopes (user + persona + persona+user)
+    - Semantic search via embeddings + keyword fallback
+
+**Automatic Extraction**
+- [ ] Background job: post-conversation analysis identifies memorable facts (opt-in)
+- [ ] Configurable extraction prompt per Persona (e.g. "extract FAQ-worthy Q&A pairs")
+- [ ] Extracted memories start in `pending_review` status
+
+**Review Pipeline**
+- [ ] Admin/team admin review queue: pending memories with approve/reject/edit
+- [ ] Persona memory review: team admins see what Personas are "learning"
+- [ ] User memory review: users see their own memories in Settings
+
+**Memory Injection**
+- [ ] At completion time: inject relevant memories into system prompt
+- [ ] Context budget aware: limit injected memory tokens
+- [ ] Scope priority: persona+user > persona > user (most specific wins on conflicts)
+
+**Use Cases**
+- [ ] Helpdesk Persona: builds FAQ from repeated questions, human-reviewable
+- [ ] Roleplay Persona: isolated memory prevents tone/context bleeding from user's other conversations
+- [ ] Tutoring Persona: tracks per-student progress, common misconceptions
+- [ ] Sales Persona: accumulates objection responses from real conversations
+- [ ] Onboarding Persona: learns what new hires commonly struggle with
+
+**User Controls**
+- [ ] Settings → Memory: view, edit, delete personal memories
+- [ ] Per-Persona memory toggle: "Allow this persona to remember things about me"
+- [ ] Admin: enable/disable, retention policies, per-team scoping
+- [ ] Privacy: user memories never cross team boundaries; persona memories respect group access
 
 ---
 
-## v0.17.0 — @mention Routing + Multi-model
+## v0.19.0 — Projects / Workspaces
+
+Organizational containers that group related conversations, KBs, and
+notes into a single workspace with shared access controls.
+
+Depends on: user groups (v0.16.0), knowledge bases (v0.14.0).
+
+**Data Model**
+- [ ] `projects` table: `id`, `name`, `description`, `scope` (personal, team, global), `owner_id`, `team_id`, `settings` (JSONB), `created_at`, `updated_at`
+- [ ] `project_resources` table: `project_id`, `resource_type` (channel, knowledge_base, note), `resource_id`, `added_at`
+- [ ] Projects use the same grant model (v0.16.0): team_only / global / groups
+
+**Features**
+- [ ] Channels can belong to a project (optional `project_id` FK)
+- [ ] KBs can belong to a project → auto-linked to all channels in the project
+- [ ] Notes can belong to a project → visible to all project members
+- [ ] Project-level Persona default: all new channels in the project start with this Persona
+- [ ] Project sidebar section: grouped view of conversations with folder-like nesting
+
+**UI**
+- [ ] Sidebar: projects as collapsible groups above/integrated with chat history
+- [ ] Project creation dialog (name, description, default Persona, default KBs)
+- [ ] Drag channels/notes into projects
+- [ ] Folders within projects (lightweight — just a `path` column on project_resources)
+- [ ] Project settings: manage resources, access, defaults
+
+**Enterprise Use**
+- [ ] Team admins create projects for their teams
+- [ ] Project templates: pre-configured with specific Personas, KBs, and settings
+- [ ] Cross-team projects via group grants
+
+---
+
+## v0.20.0 — @mention Routing + Multi-model
 
 The channel schema already supports multiple models. This adds routing.
 
+_(Shifted from v0.17.0 — no content changes)_
+
 - [ ] @mention parsing in messages (users and AI models)
 - [ ] Resolve mentions against `channel_models`
 - [ ] Multi-model channels: fire completions per mentioned model
@@ -774,11 +978,13 @@ The channel schema already supports multiple models. This adds routing.
 
 ---
 
-## v0.18.0 — Extension Surfaces + Modes
+## v0.21.0 — Extension Surfaces + Modes
 
 Depends on: extension foundation (v0.11.0).
 See [EXTENSIONS.md §6](EXTENSIONS.md#6-surfaces-modes).
 
+_(Shifted from v0.18.0 — no content changes)_
+
 - [ ] Surface registration and activation API
 - [ ] Mode selector in sidebar (below brand, above chat history)
 - [ ] `ctx.ui.replace()` / `ctx.ui.restore()` for region management
@@ -788,11 +994,13 @@ See [EXTENSIONS.md §6](EXTENSIONS.md#6-surfaces-modes).
 
 ---
 
-## v0.19.0 — Smart Routing
+## v0.22.0 — Smart Routing
 
 Depends on: model roles (v0.10.0), usage tracking (v0.10.0).
 Rules-based routing engine — policy, not ML.
 
+_(Shifted from v0.19.0 — no content changes)_
+
 - [ ] Admin routing policies: "cheapest model with required capabilities",
       "prefer private providers, fallback to cloud", "for team X, use Y"
 - [ ] Fallback chains: primary provider down → next with same model
@@ -801,13 +1009,15 @@ Rules-based routing engine — policy, not ML.
 
 ---
 
-## v0.20.0 — Multi-Participant Channels + Presence
+## v0.23.0 — Multi-Participant Channels + Presence
 
 The channel foundation for workflows and real-time collaboration. Today
 channels are single-user direct chats. This release makes channels a
 shared space that multiple actors can inhabit.
 
-Depends on: extension surfaces (v0.18.0), WebSocket infrastructure (already exists).
+Depends on: extension surfaces (v0.21.0), WebSocket infrastructure (already exists).
+
+_(Shifted from v0.20.0 — no content changes)_
 
 **Channel Participants**
 - [ ] `channel_participants` table: `channel_id`, `participant_type` (user, session, persona), `participant_id`, `role` (owner, member, observer), `joined_at`
@@ -828,11 +1038,13 @@ Depends on: extension surfaces (v0.18.0), WebSocket infrastructure (already exis
 
 ---
 
-## v0.21.0 — Auth Strategy (mTLS/OIDC) + Full RBAC
+## v0.24.0 — Auth Strategy (mTLS/OIDC) + Full RBAC
 
-Enterprise auth modes on top of the teams foundation.
+Enterprise auth modes on top of the teams + groups foundation.
 Depends on: vault passphrase support from v0.9.3 (conditional unlock for
-personal providers under external auth).
+personal providers under external auth), user groups (v0.16.0).
+
+_(Shifted from v0.21.0 — enhanced with group-based permissions)_
 
 - [ ] `AUTH_MODE` env var: `builtin` | `mtls` | `oidc`
 - [ ] All three resolve to the same internal user model
@@ -842,6 +1054,8 @@ personal providers under external auth).
 - [ ] Per-source auto-activate policy: auto_activate, default_team, default_role
 - [ ] Vault passphrase prompt for mTLS/OIDC users (v0.9.3 conditional unlock)
 - [ ] Fine-grained permissions: model access, KB write, task create, token budgets
+- [ ] Group-based permission sets: assign permissions to groups, users inherit from membership
+- [ ] OIDC claim → group mapping: external IdP groups auto-sync to internal groups
 
 **Anonymous / Session Participants**
 - [ ] Session-based identity for unauthenticated visitors (workflow intake)
@@ -852,7 +1066,7 @@ personal providers under external auth).
 
 ---
 
-## v0.22.0 — Workflow Engine
+## v0.25.0 — Workflow Engine
 
 Team-owned, stage-based process execution. The channel is the runtime,
 personas drive each stage, and the existing tool/notes infrastructure
@@ -860,7 +1074,9 @@ handles structured data collection. See [ARCHITECTURE.md — Workflow
 Architecture](ARCHITECTURE.md#workflow-architecture-future--v0210) for
 the conceptual model.
 
-Depends on: multi-participant channels (v0.20.0), anonymous identity (v0.21.0).
+Depends on: multi-participant channels (v0.23.0), anonymous identity (v0.24.0).
+
+_(Shifted from v0.22.0 — no content changes, dependency refs updated)_
 
 **Workflow Definitions**
 - [ ] `workflows` table: `team_id` (owner), `name`, `description`, `is_active`, `entry_mode` (public_link, team_internal, api)
@@ -900,10 +1116,12 @@ Depends on: multi-participant channels (v0.20.0), anonymous identity (v0.21.0).
 
 ---
 
-## v0.23.0 — Tasks / Autonomous Agents
+## v0.26.0 — Tasks / Autonomous Agents
 
 Unattended execution — workflows without a human in the loop.
-Depends on: workflow engine (v0.22.0).
+Depends on: workflow engine (v0.25.0).
+
+_(Shifted from v0.23.0 — no content changes, dependency refs updated)_
 
 - [ ] Scheduler + task runner (cron-like triggers for workflow instantiation)
 - [ ] `task_create` tool (AI can spawn sub-workflows)
@@ -943,6 +1161,23 @@ based on need.
 **UX / Multi-Seat**
 - ~~Per-chat model/preset persistence (server-side)~~ ✅ _(v0.12.0 — `channels.settings.last_selector_id` JSONB merge, localStorage write-through cache)_
 
+**Knowledge Bases — Future**
+- Hybrid search: combine vector similarity with full-text `tsvector`, re-rank
+- Semantic chunking: embedding-based boundary detection for smarter splits
+- HNSW index: better query performance than IVFFlat for large datasets
+- Web scraping source: ingest URLs as KB documents (extends url_fetch)
+- Scheduled re-indexing: periodic rebuild when source documents update
+- ~~KB sharing: cross-team access grants~~ → subsumed by User Groups (v0.16.0) + Resource Grants
+- Store cleanup: add `UpdateDocumentStorageKey()` to `KnowledgeBaseStore` interface
+  (currently uses direct `database.DB.ExecContext` in handler — works but bypasses store layer)
+
+**Memory — Future**
+- Memory compaction: summarize old memories to save context tokens
+- Memory confidence decay: reduce confidence over time, prune low-confidence entries
+- Memory export/import: portable memory format across instances
+- Cross-persona memory sharing (opt-in): e.g. "coding assistant" can read facts from "project manager"
+- Memory analytics: dashboard showing what Personas are learning, memory growth trends
+
 **Platform**
 - Rate limiting per user/team/tier (token budgets)
 - Provider health monitoring + key rotation