# DESIGN — Multi-File Starlark Packages

Promotes Starlark scripts from a single inline blob to a proper
file tree with `load()` support. Prerequisite for library packages.

---

## Current State

Three problems:

1. **Scripts live in the DB.** The installer injects `script.star`
   contents into `manifest["_starlark_script"]`. The runner reads
   the script from the manifest JSONB column at runtime. This means
   the entire script — no matter how large — is serialized into a
   JSON string inside a JSON object inside a database row.

2. **`load()` is disabled.** The sandbox callback hard-returns an
   error. There's no way to split code across files.

3. **Only static assets are extracted.** `extractableRelPath()`
   allows `js/`, `css/`, `assets/` — nothing else. Starlark files
   land in the DB, not on disk.

The hotfix in v0.37.14 (reading `script.star` from the archive and
injecting it as `_starlark_script`) made single-file clean but
didn't fix the underlying architecture.

---

## Design

### Archive Structure

```
gitea-client/
├── manifest.json          ← metadata only, no _starlark_script
├── script.star            ← entry point (required for starlark tier)
├── star/                  ← optional: additional modules
│   ├── repos.star
│   ├── issues.star
│   ├── auth.star
│   └── ci.star
├── js/                    ← optional: surface assets
│   └── main.js
└── css/
    └── main.css
```

Convention: `script.star` is always the entry point. Submodules live
in `star/`. This parallels `js/main.js` as the JS entry point with
supporting files alongside it.

### Manifest Change

The manifest gains an optional `entry_point` field for packages that
want a different entry script name. Default: `script.star`.

```json
{
  "id": "gitea-client",
  "type": "library",
  "tier": "starlark",
  "entry_point": "script.star"
}
```

`_starlark_script` is no longer written to the manifest. The field
is still read for backward compatibility (existing packages that have
it in their DB row continue to work until reinstalled).

### Extraction

**`extractableRelPath()`** adds `star/` and bare `*.star` files:

```go
var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}

func extractableRelPath(name string) string {
    // Existing prefix matching for directories...

    // Also extract bare .star files at archive root
    base := filepath.Base(name)
    if strings.HasSuffix(base, ".star") {
        // Strip leading directory (package-id/) if present
        if idx := strings.Index(name, "/"); idx >= 0 {
            rest := name[idx+1:]
            if rest == base || strings.HasPrefix(rest, "star/") {
                return rest
            }
        }
        if name == base {
            return name
        }
    }

    return ""
}
```

After install, the packages directory looks like:

```
/data/packages/gitea-client/
├── script.star
├── star/
│   ├── repos.star
│   ├── issues.star
│   ├── auth.star
│   └── ci.star
├── js/
│   └── main.js
└── css/
    └── main.css
```

### Installer Changes

**`packages.go` `InstallPackage()`:**

1. Stop injecting `_starlark_script` into the manifest. Remove the
   hotfix code that reads `script.star` from the archive and stuffs
   it into the manifest JSON.

2. Let `extractableRelPath()` handle `.star` files naturally — they
   get extracted to disk alongside `js/` and `css/`.

3. Validate: if `tier == "starlark"`, confirm the archive contains
   `script.star` (or the manifest's `entry_point` value). Return
   400 if missing.

### Runner Changes

**`runner.go` `ExecPackage()`:**

Replace manifest-based script loading with file-based:

```go
func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
    if pkg.Status != models.PackageStatusActive {
        return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
    }
    if pkg.Tier != models.ExtTierStarlark {
        return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
    }

    // ── Load script from disk (primary) or manifest (legacy) ──
    script, err := r.loadScript(pkg)
    if err != nil {
        return nil, err
    }

    modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
    if err != nil {
        return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
    }

    // Build package-scoped load callback
    loader := r.packageLoader(pkg.ID, modules)

    log.Printf("  🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))

    return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
}

func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
    // Primary: read from disk
    if r.packagesDir != "" {
        entryPoint := "script.star"
        if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
            entryPoint = ep
        }
        path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
        data, err := os.ReadFile(path)
        if err == nil && len(data) > 0 {
            return string(data), nil
        }
        // Fall through to legacy
    }

    // Legacy: inline in manifest
    script, ok := pkg.Manifest["_starlark_script"].(string)
    if ok && script != "" {
        return script, nil
    }

    return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
}
```

### Sandbox Changes

**`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but
accepts a `load` callback:

```go
// LoadFunc resolves load("path") calls to Starlark source.
// Returns the module's globals. The sandbox caches results per
// thread (Starlark handles this via thread.Load dedup).
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)

func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
    modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {

    // ... same as Exec() but thread.Load = loader instead of error ...
}
```

The existing `Exec()` becomes a thin wrapper that passes a
nil/error loader for backward compatibility.

### Package-Scoped Loader

**`runner.go`** builds the load callback scoped to the package's
directory:

```go
func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
    if r.packagesDir == "" {
        return nil // no disk = no load support
    }

    pkgDir := filepath.Join(r.packagesDir, pkgID)
    cache := make(map[string]*loadEntry) // dedup + cycle detection

    return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
        // ── Security: reject traversal ──
        if strings.Contains(module, "..") || filepath.IsAbs(module) {
            return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
        }

        // ── Resolve to package directory ──
        resolved := filepath.Join(pkgDir, module)
        if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
            return nil, fmt.Errorf("load: path escapes package directory: %q", module)
        }

        // ── Must be a .star file ──
        if !strings.HasSuffix(resolved, ".star") {
            return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
        }

        // ── Cache / cycle detection ──
        if entry, ok := cache[module]; ok {
            if entry.loading {
                return nil, fmt.Errorf("load: circular dependency: %q", module)
            }
            return entry.globals, nil
        }

        entry := &loadEntry{loading: true}
        cache[module] = entry

        // ── Read and execute ──
        data, err := os.ReadFile(resolved)
        if err != nil {
            return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
        }

        globals, err := starlark.ExecFile(thread, module, string(data),
            r.predeclaredForLoad(modules))
        if err != nil {
            return nil, fmt.Errorf("load: error in %q: %w", module, err)
        }

        entry.globals = globals
        entry.loading = false
        return globals, nil
    }
}

type loadEntry struct {
    loading bool
    globals starlark.StringDict
}
```

Key constraints:

- **Package-scoped.** Can only load files from within the package's
  own directory. Path traversal (`..`) and absolute paths are rejected.
  One package cannot load another package's files.

- **`.star` files only.** Cannot load `.js`, `.json`, or anything
  else. This is a Starlark sandbox, not a file reader.

- **Circular dependency detection.** If A loads B loads A → error.

- **Cached per invocation.** If `script.star` and `repos.star` both
  load `auth.star`, it executes once. Starlark's thread-level load
  dedup handles this, plus our explicit cache.

- **Same predeclared modules.** Loaded files get the same `db`,
  `http`, `json`, `settings`, `connections` modules as the entry
  point. They run in the same permission context.

### Starlark Usage

```python
# script.star (entry point)
load("star/auth.star", "auth_headers", "auth_headers_json")
load("star/repos.star", "get_repos", "sync_repos")
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
load("star/ci.star", "get_ci_status")

def on_request(req):
    # ... dispatch to imported functions ...

def on_tool_call(tool_name, params):
    # ... dispatch to imported functions ...
```

```python
# star/auth.star
def auth_headers(conn):
    return {"Authorization": "token " + conn.get("api_token", "")}

def auth_headers_json(conn):
    h = auth_headers(conn)
    h["Content-Type"] = "application/json"
    return h
```

```python
# star/repos.star
load("star/auth.star", "auth_headers")

def get_repos(conn):
    url = conn["base_url"] + "/api/v1/repos/search?limit=50"
    resp = http.get(url=url, headers=auth_headers(conn))
    if int(resp["status"]) >= 400:
        return None
    return json.decode(resp["body"])

def sync_repos(conn):
    repos = get_repos(conn)
    # ... db.insert into private tables ...
```

Submodules can load other submodules. The dependency graph is a DAG
(cycles are rejected).

### Build Script

**`packages/build.sh`** already includes `script.star`. Add `star/`:

```bash
local dirs=""
[ -d "$dir/js" ]          && dirs="$dirs js/"
[ -d "$dir/css" ]         && dirs="$dirs css/"
[ -d "$dir/assets" ]      && dirs="$dirs assets/"
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
[ -d "$dir/star" ]        && dirs="$dirs star/"
[ -d "$dir/migrations" ]  && dirs="$dirs migrations/"
```

### Runner Constructor

**`runner.go`** gains `packagesDir`:

```go
type Runner struct {
    sandbox     *Sandbox
    stores      store.Stores
    packagesDir string              // NEW
    notifier    NotificationSender
    resolver    ProviderResolver
    db          *sql.DB
    dbPostgres  bool
}

func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
    return &Runner{sandbox: sb, stores: stores}
}

func (r *Runner) SetPackagesDir(dir string) {
    r.packagesDir = dir
}
```

**`main.go`** wires it:

```go
starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
```

---

## Backward Compatibility

| Scenario | Behavior |
|----------|----------|
| Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. |
| Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. |
| New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. |
| New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. |
| Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. |
| Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". |

No migration needed. Old packages work as-is. New packages benefit
from the file-based path. Reinstalling an old package extracts its
`script.star` to disk (via the updated `extractableRelPath`).

---

## Security

| Threat | Mitigation |
|--------|------------|
| Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. |
| Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. |
| Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. |
| Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. |
| Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. |
| Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. |
| Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. |

---

## Changeset Plan

| CS | Scope | Files |
|----|-------|-------|
| CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. |
| CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. |
| CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. |
| CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. |
| CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. |

CS1-CS4 can land in a single session. CS5 validates. All five are
backend-only — no FE changes.

---

## What This Enables

- **Library packages** — a `gitea-client` with `script.star` as
  entry point, `star/repos.star`, `star/issues.star`, `star/ci.star`
  as submodules. Clean separation of concerns.

- **Large extensions** — a workflow automation package with 20+
  tool handlers doesn't have to be one 2000-line file.

- **Shared utilities** — within a single package, common helpers
  (auth, validation, formatting) live in their own file and are
  loaded by multiple entry points.

- **Testable modules** — each `.star` file can be loaded and
  tested independently in a dev sandbox.

This does NOT enable cross-package loading. Package A cannot load
Package B's files. That's what `lib.load()` is for (see
DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is
intra-package. `lib.load()` is inter-package. Different mechanisms,
different trust boundaries.