# DESIGN — Extension Connections & Library Packages Two platform primitives. Extensions that talk to external services need scoped credential management. Extensions that share logic and data need a dependency mechanism with clean boundaries. --- ## Part 1: Extension Connections ### Problem Extensions integrating with external services need credentials and endpoint config. The current `settings` mechanism is flat key-value. It breaks when a user works with two instances of the same service, when teams share bot tokens alongside personal tokens, or when multiple extensions need the same credentials. ### Design A **connection** concept. Same scope/resolution pattern as LLM providers — scoped CRUD, resolution chain — but generic, owned by package manifests, and sharable across packages. #### Manifest Declaration ```json { "id": "git-board", "connections": [ { "type": "gitea", "label": "Gitea Instance", "fields": { "base_url": {"type": "url", "required": true, "label": "Server URL"}, "api_token": {"type": "secret", "required": true, "label": "API Token"}, "org": {"type": "string", "required": false, "label": "Default Org"} }, "scopes": ["global", "team", "personal"] }, { "type": "github", "label": "GitHub", "fields": { "api_token": {"type": "secret", "required": true, "label": "Personal Access Token"}, "org": {"type": "string", "required": false, "label": "Default Org"} }, "scopes": ["global", "personal"] } ] } ``` | Field | Description | |-------|-------------| | `type` | Connection type identifier. Shared by name — if two packages declare `type: "gitea"`, users configure once and both packages resolve it. Prefix to namespace: `"git-board:internal"`. | | `label` | Human-readable name for the UI. | | `fields` | Config form schema. Types: `string`, `url`, `secret`, `number`, `boolean`, `select`. | | `scopes` | Allowed scopes. Subset of `["global", "team", "personal"]`. | #### Storage ```sql CREATE TABLE ext_connections ( id TEXT PRIMARY KEY, type TEXT NOT NULL, package_id TEXT NOT NULL, -- declaring package (UI attribution) scope TEXT NOT NULL, -- global | team | personal owner_id TEXT NOT NULL, -- '' for global, team_id, or user_id name TEXT NOT NULL, -- user label: "Work Gitea" config TEXT NOT NULL, -- JSON, secrets encrypted at rest is_active BOOLEAN DEFAULT TRUE, created_at TIMESTAMP, updated_at TIMESTAMP, UNIQUE(type, scope, owner_id, name) ); ``` Secrets use the same vault pattern as `provider_configs.api_key`. #### Resolution Chain Personal → Team → Global. Same walk as provider resolution. ```python conn = connections.get("gitea") # scope chain conn = connections.get("gitea", name="Work Gitea") # explicit all = connections.list("gitea") # all available ``` #### API Endpoints ``` # Admin (global) POST/GET/PUT/DELETE /api/v1/admin/connections[/:id] # Team POST/GET/PUT/DELETE /api/v1/teams/:teamId/connections[/:id] # Personal POST/GET/PUT/DELETE /api/v1/connections[/:id] # Resolution GET /api/v1/connections/resolve?type=gitea&name=Work+Gitea ``` #### UI Three surfaces matching the provider config pattern: - **Admin → Connections** — global. Type picker from installed packages. - **Team Admin → Connections** — team-scoped. - **Settings → Connections** — personal. --- ## Part 2: Library Packages ### Problem Extensions duplicate everything. Two packages talking to Gitea write the same API client, the same auth logic, the same pagination. If they cache data, each creates its own tables with the same schema. There's no mechanism for sharing code, data access, or connection types across packages. ### Core Principle **Libraries are services, not shared databases.** A library owns its data privately. Consumers access it through exported Starlark functions or REST endpoints — never by querying the library's tables directly. The library controls validation, access patterns, schema evolution, and data integrity. Consumers are decoupled from the physical storage. Same principle as microservices vs shared databases: share the schema, share the coupling. Share the API, share the contract. ### Design A new package type: `library`. Libraries can provide: - **Starlark functions** — server-side consumers call via `lib.load()` - **REST endpoints** — browser-side consumers call via HTTP - **Connection types** — inherited by consumers - **Private DB tables** — optional, never exposed directly - Any combination of the above Libraries do **not** have: surfaces, LLM tools, or pipe filters. #### Package Type Taxonomy | Type | Surface | Starlark exports | REST endpoints | LLM tools | Private DB | Depends on libs | |------|---------|-----------------|----------------|-----------|------------|-----------------| | `surface` | ✓ | — | — | — | — | via REST | | `extension` | — | — | — | ✓ | ✓ | ✓ | | `library` | — | ✓ | ✓ | — | ✓ | ✓ | | `full` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | #### Library Manifest ```json { "id": "gitea-client", "title": "Gitea API Client", "type": "library", "tier": "starlark", "version": "1.0.0", "description": "Shared Gitea client — connections, API, and data caching.", "permissions": ["api.http", "db.read", "db.write"], "exports": [ "get_repos", "sync_repos", "get_issues", "get_issue", "create_issue", "update_issue", "get_prs", "get_ci_status", "search_cached_issues" ], "api_routes": [ {"method": "GET", "path": "/repos"}, {"method": "GET", "path": "/issues"}, {"method": "GET", "path": "/issues/*"}, {"method": "POST", "path": "/issues"}, {"method": "GET", "path": "/prs"}, {"method": "GET", "path": "/ci/*"}, {"method": "POST", "path": "/sync"} ], "connections": [ { "type": "gitea", "label": "Gitea Instance", "fields": { "base_url": {"type": "url", "required": true}, "api_token": {"type": "secret", "required": true}, "org": {"type": "string", "required": false} }, "scopes": ["global", "team", "personal"] } ], "db_tables": [ { "name": "repos", "columns": { "connection_id": "text", "full_name": "text", "owner": "text", "name": "text", "description": "text", "html_url": "text", "open_issues": "integer", "synced_at": "text" } }, { "name": "issues", "columns": { "connection_id": "text", "repo_full_name": "text", "number": "integer", "title": "text", "state": "text", "labels": "text", "assignee": "text", "body": "text", "created_at": "text", "synced_at": "text" } } ], "schema_version": 1 } ``` Physical tables: `ext_gitea_client_repos`, `ext_gitea_client_issues`. Private to the library. The db module remains package-scoped. `physicalTable()` is unchanged — no cross-package table access. #### script.star (library) Two entry points. `exports` for Starlark consumers. `on_request` for REST consumers. Both use the same internal logic. The DB is an implementation detail. ```python # ═══════════════════════════════════════════ # Exported functions (Starlark consumers) # ═══════════════════════════════════════════ def get_repos(conn): """Fetch repos live from Gitea API.""" url = conn["base_url"] + "/api/v1/repos/search?limit=50" resp = http.get(url=url, headers=_auth(conn)) if int(resp["status"]) >= 400: return None return json.decode(resp["body"]) def sync_repos(conn): """Fetch repos and cache locally. Returns count.""" repos = get_repos(conn) if repos == None: return 0 conn_id = conn.get("id", "") for row in db.query("repos", filters={"connection_id": conn_id}): db.delete("repos", row["id"]) for r in repos: db.insert("repos", { "connection_id": conn_id, "full_name": r.get("full_name", ""), "owner": r.get("owner", {}).get("login", ""), "name": r.get("name", ""), "description": r.get("description", ""), "html_url": r.get("html_url", ""), "open_issues": int(r.get("open_issues_count", 0)), "synced_at": "", }) return len(repos) def get_issues(conn, owner, repo, state="open"): """Fetch issues live from Gitea API.""" url = (conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo + "/issues?state=" + state + "&limit=50&type=issues") resp = http.get(url=url, headers=_auth(conn)) if int(resp["status"]) >= 400: return None return json.decode(resp["body"]) def search_cached_issues(conn_id, repo_full_name, state=None): """Query locally cached issues. No API call.""" filters = {"connection_id": conn_id, "repo_full_name": repo_full_name} if state: filters["state"] = state return db.query("issues", filters=filters, order="-synced_at", limit=200) def create_issue(conn, owner, repo, title, body="", labels=None): """Create issue via Gitea API.""" payload = {"title": title} if body: payload["body"] = body if labels: payload["labels"] = labels url = conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo + "/issues" resp = http.post(url=url, body=json.encode(payload), headers=_auth_json(conn)) if int(resp["status"]) >= 400: return None return json.decode(resp["body"]) # ... update_issue, get_prs, get_ci_status same pattern ... # ═══════════════════════════════════════════ # REST endpoints (browser / external) # ═══════════════════════════════════════════ def on_request(req): path = req["path"] method = req["method"] conn = connections.get("gitea") if not conn: return _resp(400, {"error": "no gitea connection configured"}) if method == "GET" and path == "/repos": return _resp(200, {"data": get_repos(conn) or []}) if method == "POST" and path == "/sync": return _resp(200, {"synced": sync_repos(conn)}) if method == "GET" and path == "/issues": q = req.get("query", {}) owner = _str(q.get("owner", "")) repo = _str(q.get("repo", "")) if owner and repo: return _resp(200, {"data": get_issues(conn, owner, repo) or []}) return _resp(200, {"data": db.query("issues", order="-synced_at", limit=200)}) return _resp(404, {"error": "not found"}) def _auth(conn): return {"Authorization": "token " + conn.get("api_token", "")} def _auth_json(conn): return {"Authorization": "token " + conn.get("api_token", ""), "Content-Type": "application/json"} def _resp(status, data): return {"status": status, "body": json.encode(data), "headers": {"Content-Type": "application/json"}} def _str(v): return str(v) if v != None else "" ``` The DB calls are all in the library's script. Consumers never see the physical tables. #### Consumer: Starlark Path ```json { "id": "ci-monitor", "type": "extension", "tier": "starlark", "dependencies": {"gitea-client": ">=1.0.0"}, "tools": [{"name": "check_ci", "description": "Check CI status", "parameters": {"owner": {"type": "string", "required": true}, "repo": {"type": "string", "required": true}, "ref": {"type": "string", "required": true}}}] } ``` ```python gitea = lib.load("gitea-client") def on_tool_call(tool_name, params): conn = connections.get("gitea") if tool_name == "check_ci": return gitea.get_ci_status(conn, params["owner"], params["repo"], params["ref"]) return {"error": "unknown tool"} ``` No `db_tables`. No `connections` declaration. No `permissions` for db access. The consumer calls functions. #### Consumer: REST Path (browser-only) A pure `type: "surface"` package calls the library's REST endpoints: ```js // git-board/js/main.js — no Starlark, no lib.load var repos = await fetch(BASE + '/s/gitea-client/api/repos', { headers: { 'Authorization': 'Bearer ' + token } }).then(r => r.json()); await fetch(BASE + '/s/gitea-client/api/sync', { method: 'POST', headers: { 'Authorization': 'Bearer ' + token } }); ``` This means `git-board` can be `type: "surface"` (browser-only). Tools come from the library. Data comes from the library's REST API. Connections come from the library. The surface just renders. #### Permission Model Libraries have their **own** permissions, granted by the admin: - `gitea-client` gets `api.http` + `db.read` + `db.write` - `ci-monitor` gets nothing beyond its own tools When `ci-monitor` calls `gitea.get_ci_status(conn, ...)`, the library function runs with the **library's** permissions. The library can make HTTP calls and write to its own tables regardless of what the consumer has. The admin trusts the library (by granting permissions). The consumer trusts the library (by declaring a dependency). The library trusts nobody — it validates inputs and controls its own data. #### `lib.load()` Mechanics 1. Check `ext_dependencies`: caller must declare dependency. 2. Load library's `script.star`. 3. Build library's module set using **library's own** permissions. 4. Execute in sandboxed context. 5. Extract globals listed in `exports`. 6. Return as frozen `starlarkstruct.Struct`. 7. Cache per sandbox invocation. The library's `db` module resolves to `ext_gitea_client_*`. The consumer's `db` module (if any) resolves to `ext_ci_monitor_*`. No cross-contamination. `physicalTable()` unchanged. #### Dependencies ```sql CREATE TABLE ext_dependencies ( consumer_id TEXT NOT NULL, library_id TEXT NOT NULL, version_spec TEXT NOT NULL, resolved_ver TEXT NOT NULL, PRIMARY KEY (consumer_id, library_id), FOREIGN KEY (consumer_id) REFERENCES package_registry(id) ON DELETE CASCADE, FOREIGN KEY (library_id) REFERENCES package_registry(id) ON DELETE RESTRICT ); ``` `ON DELETE RESTRICT`: cannot uninstall a library with consumers. Libraries can depend on libraries. Circular deps rejected at install. #### Schema Migrations Libraries own their migrations. Consumers are unaffected. If a migration changes internal tables, the library's exported functions adapt. Private schema is NOT part of the API surface. #### Breaking Changes **API surface = exports + REST endpoints + connection types.** - Remove/rename an export or endpoint → major version bump. - Change a function's return shape → major version bump. - Change connection type fields → major version bump. - Add new exports, endpoints, fields → minor version bump. - Internal DB changes, bug fixes → patch version bump. Enforcement deferred. Social contract for now. --- ## The Full Stack ``` ┌──────────────────────────────────────────────┐ │ gitea-client (library) │ │ │ │ connections: gitea │ │ exports: get_repos, sync_repos, ... │ │ REST: /repos, /issues, /ci/*, /sync │ │ private DB: repos, issues (untouchable) │ │ │ │ permissions: api.http, db.read, db.write │ └────────┬────────────────┬────────────────┬───┘ │ │ │ lib.load() lib.load() fetch() │ │ │ ┌─────┴──────┐ ┌──────┴───────┐ ┌────┴──────┐ │ ci-monitor │ │ code-review │ │ git-board │ │ extension │ │ extension │ │ surface │ │ (starlark) │ │ (starlark) │ │ (browser) │ └────────────┘ └──────────────┘ └───────────┘ ``` One library. N consumers. Zero schema coupling. --- ## Implementation Phases ### Phase 1: Extension Connections Prerequisite: none. - `ext_connections` table + store + handlers + UI - `connections` Starlark module - Three management surfaces (admin / team / personal) ### Phase 2: Library Packages Prerequisite: none. Independent of Phase 1. - `library` package type in installer - `ext_dependencies` table - `lib.load()` with per-library permission context - Library `api_routes` (already works) - Dependency resolution + uninstall protection - Admin UI: dependency tree ### Phase 3: Full Composition Prerequisite: Phase 1 + 2. - Libraries declare connection types for consumers - Build `gitea-client` reference library - Migrate `git-board` to library-backed surface --- ## Not Covered - **OAuth flows.** Static credentials only. - **Package registry / marketplace.** `.pkg` archives, no central registry. - **Runtime version pinning.** Installed version wins. - **Cross-package events.** Functions and REST, not event subscriptions. - **Library UI.** No surface. Config via package settings + connections.