core/docs/DESIGN-EXT-CONNECTIONS-LIBRARIES.md

# DESIGN — Extension Connections & Library Packages

Two platform primitives. Extensions that talk to external services
need scoped credential management. Extensions that share logic and
data need a dependency mechanism with clean boundaries.

---

## Part 1: Extension Connections

### Problem

Extensions integrating with external services need credentials and
endpoint config. The current `settings` mechanism is flat key-value.
It breaks when a user works with two instances of the same service,
when teams share bot tokens alongside personal tokens, or when
multiple extensions need the same credentials.

### Design

A **connection** concept. Same scope/resolution pattern as LLM
providers — scoped CRUD, resolution chain — but generic, owned by
package manifests, and sharable across packages.

#### Manifest Declaration

```json
{
  "id": "git-board",
  "connections": [
    {
      "type": "gitea",
      "label": "Gitea Instance",
      "fields": {
        "base_url":  {"type": "url",    "required": true,  "label": "Server URL"},
        "api_token": {"type": "secret", "required": true,  "label": "API Token"},
        "org":       {"type": "string", "required": false, "label": "Default Org"}
      },
      "scopes": ["global", "team", "personal"]
    },
    {
      "type": "github",
      "label": "GitHub",
      "fields": {
        "api_token": {"type": "secret", "required": true,  "label": "Personal Access Token"},
        "org":       {"type": "string", "required": false, "label": "Default Org"}
      },
      "scopes": ["global", "personal"]
    }
  ]
}
```

| Field | Description |
|-------|-------------|
| `type` | Connection type identifier. Shared by name — if two packages declare `type: "gitea"`, users configure once and both packages resolve it. Prefix to namespace: `"git-board:internal"`. |
| `label` | Human-readable name for the UI. |
| `fields` | Config form schema. Types: `string`, `url`, `secret`, `number`, `boolean`, `select`. |
| `scopes` | Allowed scopes. Subset of `["global", "team", "personal"]`. |

#### Storage

```sql
CREATE TABLE ext_connections (
    id          TEXT PRIMARY KEY,
    type        TEXT NOT NULL,
    package_id  TEXT NOT NULL,          -- declaring package (UI attribution)
    scope       TEXT NOT NULL,          -- global | team | personal
    owner_id    TEXT NOT NULL,          -- '' for global, team_id, or user_id
    name        TEXT NOT NULL,          -- user label: "Work Gitea"
    config      TEXT NOT NULL,          -- JSON, secrets encrypted at rest
    is_active   BOOLEAN DEFAULT TRUE,
    created_at  TIMESTAMP,
    updated_at  TIMESTAMP,
    UNIQUE(type, scope, owner_id, name)
);
```

Secrets use the same vault pattern as `provider_configs.api_key`.

#### Resolution Chain

Personal → Team → Global. Same walk as provider resolution.

```python
conn = connections.get("gitea")                        # scope chain
conn = connections.get("gitea", name="Work Gitea")     # explicit
all  = connections.list("gitea")                       # all available
```

#### API Endpoints

```
# Admin (global)
POST/GET/PUT/DELETE  /api/v1/admin/connections[/:id]

# Team
POST/GET/PUT/DELETE  /api/v1/teams/:teamId/connections[/:id]

# Personal
POST/GET/PUT/DELETE  /api/v1/connections[/:id]

# Resolution
GET  /api/v1/connections/resolve?type=gitea&name=Work+Gitea
```

#### UI

Three surfaces matching the provider config pattern:
- **Admin → Connections** — global. Type picker from installed packages.
- **Team Admin → Connections** — team-scoped.
- **Settings → Connections** — personal.

---

## Part 2: Library Packages

### Problem

Extensions duplicate everything. Two packages talking to Gitea write
the same API client, the same auth logic, the same pagination. If
they cache data, each creates its own tables with the same schema.
There's no mechanism for sharing code, data access, or connection
types across packages.

### Core Principle

**Libraries are services, not shared databases.**

A library owns its data privately. Consumers access it through
exported Starlark functions or REST endpoints — never by querying
the library's tables directly. The library controls validation,
access patterns, schema evolution, and data integrity. Consumers
are decoupled from the physical storage.

Same principle as microservices vs shared databases: share the
schema, share the coupling. Share the API, share the contract.

### Design

A new package type: `library`. Libraries can provide:

- **Starlark functions** — server-side consumers call via `lib.load()`
- **REST endpoints** — browser-side consumers call via HTTP
- **Connection types** — inherited by consumers
- **Private DB tables** — optional, never exposed directly
- Any combination of the above

Libraries do **not** have: surfaces, LLM tools, or pipe filters.

#### Package Type Taxonomy

| Type | Surface | Starlark exports | REST endpoints | LLM tools | Private DB | Depends on libs |
|------|---------|-----------------|----------------|-----------|------------|-----------------|
| `surface` | ✓ | — | — | — | — | via REST |
| `extension` | — | — | — | ✓ | ✓ | ✓ |
| `library` | — | ✓ | ✓ | — | ✓ | ✓ |
| `full` | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

#### Library Manifest

```json
{
  "id": "gitea-client",
  "title": "Gitea API Client",
  "type": "library",
  "tier": "starlark",
  "version": "1.0.0",
  "description": "Shared Gitea client — connections, API, and data caching.",

  "permissions": ["api.http", "db.read", "db.write"],

  "exports": [
    "get_repos", "sync_repos",
    "get_issues", "get_issue", "create_issue", "update_issue",
    "get_prs", "get_ci_status",
    "search_cached_issues"
  ],

  "api_routes": [
    {"method": "GET",  "path": "/repos"},
    {"method": "GET",  "path": "/issues"},
    {"method": "GET",  "path": "/issues/*"},
    {"method": "POST", "path": "/issues"},
    {"method": "GET",  "path": "/prs"},
    {"method": "GET",  "path": "/ci/*"},
    {"method": "POST", "path": "/sync"}
  ],

  "connections": [
    {
      "type": "gitea",
      "label": "Gitea Instance",
      "fields": {
        "base_url":  {"type": "url",    "required": true},
        "api_token": {"type": "secret", "required": true},
        "org":       {"type": "string", "required": false}
      },
      "scopes": ["global", "team", "personal"]
    }
  ],

  "db_tables": [
    {
      "name": "repos",
      "columns": {
        "connection_id": "text", "full_name": "text",
        "owner": "text", "name": "text",
        "description": "text", "html_url": "text",
        "open_issues": "integer", "synced_at": "text"
      }
    },
    {
      "name": "issues",
      "columns": {
        "connection_id": "text", "repo_full_name": "text",
        "number": "integer", "title": "text", "state": "text",
        "labels": "text", "assignee": "text", "body": "text",
        "created_at": "text", "synced_at": "text"
      }
    }
  ],

  "schema_version": 1
}
```

Physical tables: `ext_gitea_client_repos`, `ext_gitea_client_issues`.
Private to the library. The db module remains package-scoped.
`physicalTable()` is unchanged — no cross-package table access.

#### script.star (library)

Two entry points. `exports` for Starlark consumers. `on_request`
for REST consumers. Both use the same internal logic. The DB is an
implementation detail.

```python
# ═══════════════════════════════════════════
#  Exported functions (Starlark consumers)
# ═══════════════════════════════════════════

def get_repos(conn):
    """Fetch repos live from Gitea API."""
    url = conn["base_url"] + "/api/v1/repos/search?limit=50"
    resp = http.get(url=url, headers=_auth(conn))
    if int(resp["status"]) >= 400:
        return None
    return json.decode(resp["body"])

def sync_repos(conn):
    """Fetch repos and cache locally. Returns count."""
    repos = get_repos(conn)
    if repos == None:
        return 0
    conn_id = conn.get("id", "")
    for row in db.query("repos", filters={"connection_id": conn_id}):
        db.delete("repos", row["id"])
    for r in repos:
        db.insert("repos", {
            "connection_id": conn_id,
            "full_name": r.get("full_name", ""),
            "owner": r.get("owner", {}).get("login", ""),
            "name": r.get("name", ""),
            "description": r.get("description", ""),
            "html_url": r.get("html_url", ""),
            "open_issues": int(r.get("open_issues_count", 0)),
            "synced_at": "",
        })
    return len(repos)

def get_issues(conn, owner, repo, state="open"):
    """Fetch issues live from Gitea API."""
    url = (conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo
           + "/issues?state=" + state + "&limit=50&type=issues")
    resp = http.get(url=url, headers=_auth(conn))
    if int(resp["status"]) >= 400:
        return None
    return json.decode(resp["body"])

def search_cached_issues(conn_id, repo_full_name, state=None):
    """Query locally cached issues. No API call."""
    filters = {"connection_id": conn_id, "repo_full_name": repo_full_name}
    if state:
        filters["state"] = state
    return db.query("issues", filters=filters, order="-synced_at", limit=200)

def create_issue(conn, owner, repo, title, body="", labels=None):
    """Create issue via Gitea API."""
    payload = {"title": title}
    if body:
        payload["body"] = body
    if labels:
        payload["labels"] = labels
    url = conn["base_url"] + "/api/v1/repos/" + owner + "/" + repo + "/issues"
    resp = http.post(url=url, body=json.encode(payload),
                     headers=_auth_json(conn))
    if int(resp["status"]) >= 400:
        return None
    return json.decode(resp["body"])

# ... update_issue, get_prs, get_ci_status same pattern ...


# ═══════════════════════════════════════════
#  REST endpoints (browser / external)
# ═══════════════════════════════════════════

def on_request(req):
    path   = req["path"]
    method = req["method"]
    conn   = connections.get("gitea")
    if not conn:
        return _resp(400, {"error": "no gitea connection configured"})

    if method == "GET" and path == "/repos":
        return _resp(200, {"data": get_repos(conn) or []})

    if method == "POST" and path == "/sync":
        return _resp(200, {"synced": sync_repos(conn)})

    if method == "GET" and path == "/issues":
        q = req.get("query", {})
        owner = _str(q.get("owner", ""))
        repo  = _str(q.get("repo", ""))
        if owner and repo:
            return _resp(200, {"data": get_issues(conn, owner, repo) or []})
        return _resp(200, {"data": db.query("issues", order="-synced_at", limit=200)})

    return _resp(404, {"error": "not found"})

def _auth(conn):
    return {"Authorization": "token " + conn.get("api_token", "")}

def _auth_json(conn):
    return {"Authorization": "token " + conn.get("api_token", ""),
            "Content-Type": "application/json"}

def _resp(status, data):
    return {"status": status, "body": json.encode(data),
            "headers": {"Content-Type": "application/json"}}

def _str(v):
    return str(v) if v != None else ""
```

The DB calls are all in the library's script. Consumers never see
the physical tables.

#### Consumer: Starlark Path

```json
{
  "id": "ci-monitor",
  "type": "extension",
  "tier": "starlark",
  "dependencies": {"gitea-client": ">=1.0.0"},
  "tools": [{"name": "check_ci", "description": "Check CI status",
             "parameters": {"owner": {"type": "string", "required": true},
                            "repo": {"type": "string", "required": true},
                            "ref": {"type": "string", "required": true}}}]
}
```

```python
gitea = lib.load("gitea-client")

def on_tool_call(tool_name, params):
    conn = connections.get("gitea")
    if tool_name == "check_ci":
        return gitea.get_ci_status(conn, params["owner"],
                                   params["repo"], params["ref"])
    return {"error": "unknown tool"}
```

No `db_tables`. No `connections` declaration. No `permissions` for
db access. The consumer calls functions.

#### Consumer: REST Path (browser-only)

A pure `type: "surface"` package calls the library's REST endpoints:

```js
// git-board/js/main.js — no Starlark, no lib.load
var repos = await fetch(BASE + '/s/gitea-client/api/repos', {
    headers: { 'Authorization': 'Bearer ' + token }
}).then(r => r.json());

await fetch(BASE + '/s/gitea-client/api/sync', {
    method: 'POST',
    headers: { 'Authorization': 'Bearer ' + token }
});
```

This means `git-board` can be `type: "surface"` (browser-only).
Tools come from the library. Data comes from the library's REST API.
Connections come from the library. The surface just renders.

#### Permission Model

Libraries have their **own** permissions, granted by the admin:

- `gitea-client` gets `api.http` + `db.read` + `db.write`
- `ci-monitor` gets nothing beyond its own tools

When `ci-monitor` calls `gitea.get_ci_status(conn, ...)`, the
library function runs with the **library's** permissions. The
library can make HTTP calls and write to its own tables regardless
of what the consumer has.

The admin trusts the library (by granting permissions). The consumer
trusts the library (by declaring a dependency). The library trusts
nobody — it validates inputs and controls its own data.

#### `lib.load()` Mechanics

1. Check `ext_dependencies`: caller must declare dependency.
2. Load library's `script.star`.
3. Build library's module set using **library's own** permissions.
4. Execute in sandboxed context.
5. Extract globals listed in `exports`.
6. Return as frozen `starlarkstruct.Struct`.
7. Cache per sandbox invocation.

The library's `db` module resolves to `ext_gitea_client_*`.
The consumer's `db` module (if any) resolves to `ext_ci_monitor_*`.
No cross-contamination. `physicalTable()` unchanged.

#### Dependencies

```sql
CREATE TABLE ext_dependencies (
    consumer_id   TEXT NOT NULL,
    library_id    TEXT NOT NULL,
    version_spec  TEXT NOT NULL,
    resolved_ver  TEXT NOT NULL,
    PRIMARY KEY (consumer_id, library_id),
    FOREIGN KEY (consumer_id) REFERENCES package_registry(id)
        ON DELETE CASCADE,
    FOREIGN KEY (library_id)  REFERENCES package_registry(id)
        ON DELETE RESTRICT
);
```

`ON DELETE RESTRICT`: cannot uninstall a library with consumers.
Libraries can depend on libraries. Circular deps rejected at install.

#### Schema Migrations

Libraries own their migrations. Consumers are unaffected. If a
migration changes internal tables, the library's exported functions
adapt. Private schema is NOT part of the API surface.

#### Breaking Changes

**API surface = exports + REST endpoints + connection types.**

- Remove/rename an export or endpoint → major version bump.
- Change a function's return shape → major version bump.
- Change connection type fields → major version bump.
- Add new exports, endpoints, fields → minor version bump.
- Internal DB changes, bug fixes → patch version bump.

Enforcement deferred. Social contract for now.

---

## The Full Stack

```
┌──────────────────────────────────────────────┐
│           gitea-client (library)             │
│                                              │
│  connections: gitea                          │
│  exports: get_repos, sync_repos, ...         │
│  REST: /repos, /issues, /ci/*, /sync         │
│  private DB: repos, issues  (untouchable)    │
│                                              │
│  permissions: api.http, db.read, db.write    │
└────────┬────────────────┬────────────────┬───┘
         │                │                │
   lib.load()        lib.load()        fetch()
         │                │                │
   ┌─────┴──────┐  ┌──────┴───────┐  ┌────┴──────┐
   │ ci-monitor │  │ code-review  │  │ git-board │
   │ extension  │  │ extension    │  │ surface   │
   │ (starlark) │  │ (starlark)   │  │ (browser) │
   └────────────┘  └──────────────┘  └───────────┘
```

One library. N consumers. Zero schema coupling.

---

## Implementation Phases

### Phase 1: Extension Connections

Prerequisite: none.

- `ext_connections` table + store + handlers + UI
- `connections` Starlark module
- Three management surfaces (admin / team / personal)

### Phase 2: Library Packages

Prerequisite: none. Independent of Phase 1.

- `library` package type in installer
- `ext_dependencies` table
- `lib.load()` with per-library permission context
- Library `api_routes` (already works)
- Dependency resolution + uninstall protection
- Admin UI: dependency tree

### Phase 3: Full Composition

Prerequisite: Phase 1 + 2.

- Libraries declare connection types for consumers
- Build `gitea-client` reference library
- Migrate `git-board` to library-backed surface

---

## Not Covered

- **OAuth flows.** Static credentials only.
- **Package registry / marketplace.** `.pkg` archives, no central registry.
- **Runtime version pinning.** Installed version wins.
- **Cross-package events.** Functions and REST, not event subscriptions.
- **Library UI.** No surface. Config via package settings + connections.