Changeset 0.28.0.1 (#173)
This commit is contained in:
221
docs/ICD/providers.md
Normal file
221
docs/ICD/providers.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# Providers & Routing
|
||||
|
||||
The **multi-provider** system. One or more LLM providers are configured,
|
||||
each with their own API keys, endpoints, and model catalogs. The routing
|
||||
layer decides which provider handles each request.
|
||||
|
||||
### User BYOK Provider Configs
|
||||
|
||||
Gated by `allow_user_byok` policy. Personal API keys are encrypted with
|
||||
the user's UEK (per-user encryption key, Argon2id-derived). Platform
|
||||
admins cannot recover personal keys.
|
||||
|
||||
```
|
||||
GET /api-configs → { "configs": [safeConfig objects] }
|
||||
POST /api-configs ← { "name", "provider", "endpoint", "api_key", ... }
|
||||
GET /api-configs/:id → safeConfig
|
||||
PUT /api-configs/:id ← partial update (api_key optional)
|
||||
DELETE /api-configs/:id
|
||||
```
|
||||
|
||||
`safeConfig` — API keys are **never** returned:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"name": "My OpenAI",
|
||||
"provider": "openai",
|
||||
"endpoint": "https://api.openai.com/v1",
|
||||
"model_default": "gpt-4o",
|
||||
"scope": "personal",
|
||||
"owner_id": "uuid",
|
||||
"is_active": true,
|
||||
"has_key": true,
|
||||
"config": {},
|
||||
"headers": {},
|
||||
"settings": {},
|
||||
"created_at": "..."
|
||||
}
|
||||
```
|
||||
|
||||
**List models for a user config:**
|
||||
|
||||
```
|
||||
GET /api-configs/:id/models
|
||||
```
|
||||
|
||||
Returns `{ "models": [catalog entries] }`.
|
||||
|
||||
**Fetch/sync models from provider API:**
|
||||
|
||||
```
|
||||
POST /api-configs/:id/models/fetch
|
||||
```
|
||||
|
||||
Calls the provider's model list API, upserts into the local catalog.
|
||||
|
||||
### Admin Global Provider Configs
|
||||
|
||||
```
|
||||
GET /admin/configs → { "configs": [configWithKey objects] }
|
||||
POST /admin/configs ← { "name", "provider", "endpoint", "api_key", "config", "headers", "settings", "is_private" }
|
||||
PUT /admin/configs/:id ← partial update
|
||||
DELETE /admin/configs/:id
|
||||
```
|
||||
|
||||
`configWithKey` is the same as `safeConfig` but comes from
|
||||
`ListGlobal` — still redacts API keys, just adds the `has_key` flag.
|
||||
|
||||
Admin configs use the `ENCRYPTION_KEY` env var (not per-user UEK).
|
||||
`is_private`: when true, the config is available for admin-created
|
||||
Personas but not directly selectable by users.
|
||||
|
||||
### Model Catalog (Admin)
|
||||
|
||||
The catalog is populated by fetching from provider APIs and stores
|
||||
model metadata (capabilities, context window, pricing).
|
||||
|
||||
```
|
||||
GET /admin/models → { "models": [catalog entries] }
|
||||
PUT /admin/models/:id ← { "visibility", "display_name", ... }
|
||||
PUT /admin/models/bulk ← { "provider_config_id", "visibility" }
|
||||
DELETE /admin/models/:id
|
||||
POST /admin/models/fetch ← { "provider_config_id": "uuid|empty" }
|
||||
```
|
||||
|
||||
**Fetch** with empty `provider_config_id` syncs ALL active global
|
||||
providers. Returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"message": "models synced",
|
||||
"added": 5,
|
||||
"updated": 12,
|
||||
"total": 47
|
||||
}
|
||||
```
|
||||
|
||||
Or for multi-provider fetch: `{ "added", "updated", "total", "errors": [...] }`.
|
||||
|
||||
**Bulk visibility** sets all models for a provider (or all models
|
||||
globally if no `provider_config_id`) to the specified visibility
|
||||
(`enabled`, `disabled`, `team`).
|
||||
|
||||
### Provider Health
|
||||
|
||||
Real-time health tracking per provider config. The health accumulator
|
||||
records success/failure/latency on every completion, flushes to DB
|
||||
every 60 seconds.
|
||||
|
||||
**Get all:**
|
||||
|
||||
```
|
||||
GET /admin/providers/health
|
||||
```
|
||||
|
||||
Returns `{ "data": [ProviderHealth objects] }`:
|
||||
|
||||
```json
|
||||
{
|
||||
"provider_config_id": "uuid",
|
||||
"provider_config_name": "OpenAI Production",
|
||||
"status": "healthy|degraded|down",
|
||||
"error_rate": 0.02,
|
||||
"avg_latency_ms": 450,
|
||||
"timeout_rate": 0.01,
|
||||
"rate_limit_count": 3,
|
||||
"last_check": "...",
|
||||
"last_error": "...|null"
|
||||
}
|
||||
```
|
||||
|
||||
**Get single:**
|
||||
|
||||
```
|
||||
GET /admin/providers/:id/health
|
||||
```
|
||||
|
||||
**Auto-disable:** After `PROVIDER_AUTO_DISABLE_THRESHOLD` consecutive
|
||||
"down" windows (default: 3), the provider is automatically deactivated.
|
||||
|
||||
### Capability Overrides
|
||||
|
||||
Admin can override any model capability detected by the catalog or
|
||||
heuristic layer.
|
||||
|
||||
```
|
||||
GET /admin/capability-overrides → { "data": [...] }
|
||||
GET /admin/models/:id/capabilities → capabilities for one model
|
||||
PUT /admin/models/:id/capabilities ← { "field": "value" }
|
||||
DELETE /admin/models/:id/capabilities/:overrideId
|
||||
```
|
||||
|
||||
Override fields: `supports_vision`, `supports_tools`, `supports_thinking`,
|
||||
`context_window`, `max_output_tokens`, etc.
|
||||
|
||||
### Routing Policies
|
||||
|
||||
Policy-based request routing. Evaluated after model/config resolution,
|
||||
before provider dispatch.
|
||||
|
||||
**Admin CRUD:**
|
||||
|
||||
```
|
||||
GET /admin/routing/policies → { "data": [...] }
|
||||
GET /admin/routing/policies/:id → policy object
|
||||
POST /admin/routing/policies ← { "name", "scope", "team_id", "priority", "policy_type", "config", "is_active" }
|
||||
PUT /admin/routing/policies/:id
|
||||
DELETE /admin/routing/policies/:id
|
||||
```
|
||||
|
||||
Policy object:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"name": "Prefer Anthropic",
|
||||
"scope": "global|team",
|
||||
"team_id": "uuid|null",
|
||||
"priority": 10,
|
||||
"policy_type": "provider_prefer|team_route|cost_limit|model_alias|capability_match",
|
||||
"config": {},
|
||||
"is_active": true
|
||||
}
|
||||
```
|
||||
|
||||
| Policy type | Config | Behavior |
|
||||
|------------|--------|----------|
|
||||
| `provider_prefer` | `{ "providers": ["cfg-1", "cfg-2"] }` | Ordered fallback list |
|
||||
| `team_route` | `{ "providers": ["cfg-1"] }` | Restrict team to specific providers |
|
||||
| `cost_limit` | `{ "max_cost_per_request": 0.50 }` | Heuristic cost cap |
|
||||
| `model_alias` | `{ "alias": "fast", "target_model": "...", "target_config": "..." }` | Alias → provider+model rewrite |
|
||||
| `capability_match` | `{ "require": ["tool_calling"], "prefer": "cheapest" }` | Match cheapest model with required capabilities |
|
||||
|
||||
**Dry-run test:**
|
||||
|
||||
```
|
||||
POST /admin/routing/test
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"user_id": "uuid",
|
||||
"team_id": "uuid|null"
|
||||
}
|
||||
```
|
||||
|
||||
Returns the ranked candidate list with health status for each.
|
||||
|
||||
### Provider Types
|
||||
|
||||
Registry of supported provider types with metadata.
|
||||
|
||||
```
|
||||
GET /admin/provider-types
|
||||
```
|
||||
|
||||
Returns `{ "types": [...] }` with name, display name, default endpoint,
|
||||
profile schema, and supported features per type.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user