222 lines
5.8 KiB
Markdown
222 lines
5.8 KiB
Markdown
# Providers & Routing
|
|
|
|
The **multi-provider** system. One or more LLM providers are configured,
|
|
each with their own API keys, endpoints, and model catalogs. The routing
|
|
layer decides which provider handles each request.
|
|
|
|
### User BYOK Provider Configs
|
|
|
|
Gated by `allow_user_byok` policy. Personal API keys are encrypted with
|
|
the user's UEK (per-user encryption key, Argon2id-derived). Platform
|
|
admins cannot recover personal keys.
|
|
|
|
```
|
|
GET /api-configs → { "configs": [safeConfig objects] }
|
|
POST /api-configs ← { "name", "provider", "endpoint", "api_key", ... }
|
|
GET /api-configs/:id → safeConfig
|
|
PUT /api-configs/:id ← partial update (api_key optional)
|
|
DELETE /api-configs/:id
|
|
```
|
|
|
|
`safeConfig` — API keys are **never** returned:
|
|
|
|
```json
|
|
{
|
|
"id": "uuid",
|
|
"name": "My OpenAI",
|
|
"provider": "openai",
|
|
"endpoint": "https://api.openai.com/v1",
|
|
"model_default": "gpt-4o",
|
|
"scope": "personal",
|
|
"owner_id": "uuid",
|
|
"is_active": true,
|
|
"has_key": true,
|
|
"config": {},
|
|
"headers": {},
|
|
"settings": {},
|
|
"created_at": "..."
|
|
}
|
|
```
|
|
|
|
**List models for a user config:**
|
|
|
|
```
|
|
GET /api-configs/:id/models
|
|
```
|
|
|
|
Returns `{ "models": [catalog entries] }`.
|
|
|
|
**Fetch/sync models from provider API:**
|
|
|
|
```
|
|
POST /api-configs/:id/models/fetch
|
|
```
|
|
|
|
Calls the provider's model list API, upserts into the local catalog.
|
|
|
|
### Admin Global Provider Configs
|
|
|
|
```
|
|
GET /admin/configs → { "configs": [configWithKey objects] }
|
|
POST /admin/configs ← { "name", "provider", "endpoint", "api_key", "config", "headers", "settings", "is_private" }
|
|
PUT /admin/configs/:id ← partial update
|
|
DELETE /admin/configs/:id
|
|
```
|
|
|
|
`configWithKey` is the same as `safeConfig` but comes from
|
|
`ListGlobal` — still redacts API keys, just adds the `has_key` flag.
|
|
|
|
Admin configs use the `ENCRYPTION_KEY` env var (not per-user UEK).
|
|
`is_private`: when true, the config is available for admin-created
|
|
Personas but not directly selectable by users.
|
|
|
|
### Model Catalog (Admin)
|
|
|
|
The catalog is populated by fetching from provider APIs and stores
|
|
model metadata (capabilities, context window, pricing).
|
|
|
|
```
|
|
GET /admin/models → { "models": [catalog entries] }
|
|
PUT /admin/models/:id ← { "visibility", "display_name", ... }
|
|
PUT /admin/models/bulk ← { "provider_config_id", "visibility" }
|
|
DELETE /admin/models/:id
|
|
POST /admin/models/fetch ← { "provider_config_id": "uuid|empty" }
|
|
```
|
|
|
|
**Fetch** with empty `provider_config_id` syncs ALL active global
|
|
providers. Returns:
|
|
|
|
```json
|
|
{
|
|
"message": "models synced",
|
|
"added": 5,
|
|
"updated": 12,
|
|
"total": 47
|
|
}
|
|
```
|
|
|
|
Or for multi-provider fetch: `{ "added", "updated", "total", "errors": [...] }`.
|
|
|
|
**Bulk visibility** sets all models for a provider (or all models
|
|
globally if no `provider_config_id`) to the specified visibility
|
|
(`enabled`, `disabled`, `team`).
|
|
|
|
### Provider Health
|
|
|
|
Real-time health tracking per provider config. The health accumulator
|
|
records success/failure/latency on every completion, flushes to DB
|
|
every 60 seconds.
|
|
|
|
**Get all:**
|
|
|
|
```
|
|
GET /admin/providers/health
|
|
```
|
|
|
|
Returns `{ "data": [ProviderHealth objects] }`:
|
|
|
|
```json
|
|
{
|
|
"provider_config_id": "uuid",
|
|
"provider_config_name": "OpenAI Production",
|
|
"status": "healthy|degraded|down",
|
|
"error_rate": 0.02,
|
|
"avg_latency_ms": 450,
|
|
"timeout_rate": 0.01,
|
|
"rate_limit_count": 3,
|
|
"last_check": "...",
|
|
"last_error": "...|null"
|
|
}
|
|
```
|
|
|
|
**Get single:**
|
|
|
|
```
|
|
GET /admin/providers/:id/health
|
|
```
|
|
|
|
**Auto-disable:** After `PROVIDER_AUTO_DISABLE_THRESHOLD` consecutive
|
|
"down" windows (default: 3), the provider is automatically deactivated.
|
|
|
|
### Capability Overrides
|
|
|
|
Admin can override any model capability detected by the catalog or
|
|
heuristic layer.
|
|
|
|
```
|
|
GET /admin/capability-overrides → { "data": [...] }
|
|
GET /admin/models/:id/capabilities → capabilities for one model
|
|
PUT /admin/models/:id/capabilities ← { "field": "value" }
|
|
DELETE /admin/models/:id/capabilities/:overrideId
|
|
```
|
|
|
|
Override fields: `supports_vision`, `supports_tools`, `supports_thinking`,
|
|
`context_window`, `max_output_tokens`, etc.
|
|
|
|
### Routing Policies
|
|
|
|
Policy-based request routing. Evaluated after model/config resolution,
|
|
before provider dispatch.
|
|
|
|
**Admin CRUD:**
|
|
|
|
```
|
|
GET /admin/routing/policies → { "data": [...] }
|
|
GET /admin/routing/policies/:id → policy object
|
|
POST /admin/routing/policies ← { "name", "scope", "team_id", "priority", "policy_type", "config", "is_active" }
|
|
PUT /admin/routing/policies/:id
|
|
DELETE /admin/routing/policies/:id
|
|
```
|
|
|
|
Policy object:
|
|
|
|
```json
|
|
{
|
|
"id": "uuid",
|
|
"name": "Prefer Anthropic",
|
|
"scope": "global|team",
|
|
"team_id": "uuid|null",
|
|
"priority": 10,
|
|
"policy_type": "provider_prefer|team_route|cost_limit|model_alias|capability_match",
|
|
"config": {},
|
|
"is_active": true
|
|
}
|
|
```
|
|
|
|
| Policy type | Config | Behavior |
|
|
|------------|--------|----------|
|
|
| `provider_prefer` | `{ "providers": ["cfg-1", "cfg-2"] }` | Ordered fallback list |
|
|
| `team_route` | `{ "providers": ["cfg-1"] }` | Restrict team to specific providers |
|
|
| `cost_limit` | `{ "max_cost_per_request": 0.50 }` | Heuristic cost cap |
|
|
| `model_alias` | `{ "alias": "fast", "target_model": "...", "target_config": "..." }` | Alias → provider+model rewrite |
|
|
| `capability_match` | `{ "require": ["tool_calling"], "prefer": "cheapest" }` | Match cheapest model with required capabilities |
|
|
|
|
**Dry-run test:**
|
|
|
|
```
|
|
POST /admin/routing/test
|
|
```
|
|
|
|
```json
|
|
{
|
|
"model": "claude-sonnet-4-20250514",
|
|
"user_id": "uuid",
|
|
"team_id": "uuid|null"
|
|
}
|
|
```
|
|
|
|
Returns the ranked candidate list with health status for each.
|
|
|
|
### Provider Types
|
|
|
|
Registry of supported provider types with metadata.
|
|
|
|
```
|
|
GET /admin/provider-types
|
|
```
|
|
|
|
Returns `{ "types": [...] }` with name, display name, default endpoint,
|
|
profile schema, and supported features per type.
|
|
|
|
---
|