456 lines
15 KiB
Markdown
456 lines
15 KiB
Markdown
# DESIGN — Multi-File Starlark Packages
|
|
|
|
Promotes Starlark scripts from a single inline blob to a proper
|
|
file tree with `load()` support. Prerequisite for library packages.
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
Three problems:
|
|
|
|
1. **Scripts live in the DB.** The installer injects `script.star`
|
|
contents into `manifest["_starlark_script"]`. The runner reads
|
|
the script from the manifest JSONB column at runtime. This means
|
|
the entire script — no matter how large — is serialized into a
|
|
JSON string inside a JSON object inside a database row.
|
|
|
|
2. **`load()` is disabled.** The sandbox callback hard-returns an
|
|
error. There's no way to split code across files.
|
|
|
|
3. **Only static assets are extracted.** `extractableRelPath()`
|
|
allows `js/`, `css/`, `assets/` — nothing else. Starlark files
|
|
land in the DB, not on disk.
|
|
|
|
The hotfix in v0.37.14 (reading `script.star` from the archive and
|
|
injecting it as `_starlark_script`) made single-file clean but
|
|
didn't fix the underlying architecture.
|
|
|
|
---
|
|
|
|
## Design
|
|
|
|
### Archive Structure
|
|
|
|
```
|
|
gitea-client/
|
|
├── manifest.json ← metadata only, no _starlark_script
|
|
├── script.star ← entry point (required for starlark tier)
|
|
├── star/ ← optional: additional modules
|
|
│ ├── repos.star
|
|
│ ├── issues.star
|
|
│ ├── auth.star
|
|
│ └── ci.star
|
|
├── js/ ← optional: surface assets
|
|
│ └── main.js
|
|
└── css/
|
|
└── main.css
|
|
```
|
|
|
|
Convention: `script.star` is always the entry point. Submodules live
|
|
in `star/`. This parallels `js/main.js` as the JS entry point with
|
|
supporting files alongside it.
|
|
|
|
### Manifest Change
|
|
|
|
The manifest gains an optional `entry_point` field for packages that
|
|
want a different entry script name. Default: `script.star`.
|
|
|
|
```json
|
|
{
|
|
"id": "gitea-client",
|
|
"type": "library",
|
|
"tier": "starlark",
|
|
"entry_point": "script.star"
|
|
}
|
|
```
|
|
|
|
`_starlark_script` is no longer written to the manifest. The field
|
|
is still read for backward compatibility (existing packages that have
|
|
it in their DB row continue to work until reinstalled).
|
|
|
|
### Extraction
|
|
|
|
**`extractableRelPath()`** adds `star/` and bare `*.star` files:
|
|
|
|
```go
|
|
var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}
|
|
|
|
func extractableRelPath(name string) string {
|
|
// Existing prefix matching for directories...
|
|
|
|
// Also extract bare .star files at archive root
|
|
base := filepath.Base(name)
|
|
if strings.HasSuffix(base, ".star") {
|
|
// Strip leading directory (package-id/) if present
|
|
if idx := strings.Index(name, "/"); idx >= 0 {
|
|
rest := name[idx+1:]
|
|
if rest == base || strings.HasPrefix(rest, "star/") {
|
|
return rest
|
|
}
|
|
}
|
|
if name == base {
|
|
return name
|
|
}
|
|
}
|
|
|
|
return ""
|
|
}
|
|
```
|
|
|
|
After install, the packages directory looks like:
|
|
|
|
```
|
|
/data/packages/gitea-client/
|
|
├── script.star
|
|
├── star/
|
|
│ ├── repos.star
|
|
│ ├── issues.star
|
|
│ ├── auth.star
|
|
│ └── ci.star
|
|
├── js/
|
|
│ └── main.js
|
|
└── css/
|
|
└── main.css
|
|
```
|
|
|
|
### Installer Changes
|
|
|
|
**`packages.go` `InstallPackage()`:**
|
|
|
|
1. Stop injecting `_starlark_script` into the manifest. Remove the
|
|
hotfix code that reads `script.star` from the archive and stuffs
|
|
it into the manifest JSON.
|
|
|
|
2. Let `extractableRelPath()` handle `.star` files naturally — they
|
|
get extracted to disk alongside `js/` and `css/`.
|
|
|
|
3. Validate: if `tier == "starlark"`, confirm the archive contains
|
|
`script.star` (or the manifest's `entry_point` value). Return
|
|
400 if missing.
|
|
|
|
### Runner Changes
|
|
|
|
**`runner.go` `ExecPackage()`:**
|
|
|
|
Replace manifest-based script loading with file-based:
|
|
|
|
```go
|
|
func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
|
|
if pkg.Status != models.PackageStatusActive {
|
|
return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
|
|
}
|
|
if pkg.Tier != models.ExtTierStarlark {
|
|
return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
|
|
}
|
|
|
|
// ── Load script from disk (primary) or manifest (legacy) ──
|
|
script, err := r.loadScript(pkg)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
|
|
}
|
|
|
|
// Build package-scoped load callback
|
|
loader := r.packageLoader(pkg.ID, modules)
|
|
|
|
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
|
|
|
|
return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
|
|
}
|
|
|
|
func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
|
|
// Primary: read from disk
|
|
if r.packagesDir != "" {
|
|
entryPoint := "script.star"
|
|
if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
|
|
entryPoint = ep
|
|
}
|
|
path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
|
|
data, err := os.ReadFile(path)
|
|
if err == nil && len(data) > 0 {
|
|
return string(data), nil
|
|
}
|
|
// Fall through to legacy
|
|
}
|
|
|
|
// Legacy: inline in manifest
|
|
script, ok := pkg.Manifest["_starlark_script"].(string)
|
|
if ok && script != "" {
|
|
return script, nil
|
|
}
|
|
|
|
return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
|
|
}
|
|
```
|
|
|
|
### Sandbox Changes
|
|
|
|
**`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but
|
|
accepts a `load` callback:
|
|
|
|
```go
|
|
// LoadFunc resolves load("path") calls to Starlark source.
|
|
// Returns the module's globals. The sandbox caches results per
|
|
// thread (Starlark handles this via thread.Load dedup).
|
|
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)
|
|
|
|
func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
|
|
modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {
|
|
|
|
// ... same as Exec() but thread.Load = loader instead of error ...
|
|
}
|
|
```
|
|
|
|
The existing `Exec()` becomes a thin wrapper that passes a
|
|
nil/error loader for backward compatibility.
|
|
|
|
### Package-Scoped Loader
|
|
|
|
**`runner.go`** builds the load callback scoped to the package's
|
|
directory:
|
|
|
|
```go
|
|
func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
|
|
if r.packagesDir == "" {
|
|
return nil // no disk = no load support
|
|
}
|
|
|
|
pkgDir := filepath.Join(r.packagesDir, pkgID)
|
|
cache := make(map[string]*loadEntry) // dedup + cycle detection
|
|
|
|
return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
|
|
// ── Security: reject traversal ──
|
|
if strings.Contains(module, "..") || filepath.IsAbs(module) {
|
|
return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
|
|
}
|
|
|
|
// ── Resolve to package directory ──
|
|
resolved := filepath.Join(pkgDir, module)
|
|
if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
|
|
return nil, fmt.Errorf("load: path escapes package directory: %q", module)
|
|
}
|
|
|
|
// ── Must be a .star file ──
|
|
if !strings.HasSuffix(resolved, ".star") {
|
|
return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
|
|
}
|
|
|
|
// ── Cache / cycle detection ──
|
|
if entry, ok := cache[module]; ok {
|
|
if entry.loading {
|
|
return nil, fmt.Errorf("load: circular dependency: %q", module)
|
|
}
|
|
return entry.globals, nil
|
|
}
|
|
|
|
entry := &loadEntry{loading: true}
|
|
cache[module] = entry
|
|
|
|
// ── Read and execute ──
|
|
data, err := os.ReadFile(resolved)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
|
|
}
|
|
|
|
globals, err := starlark.ExecFile(thread, module, string(data),
|
|
r.predeclaredForLoad(modules))
|
|
if err != nil {
|
|
return nil, fmt.Errorf("load: error in %q: %w", module, err)
|
|
}
|
|
|
|
entry.globals = globals
|
|
entry.loading = false
|
|
return globals, nil
|
|
}
|
|
}
|
|
|
|
type loadEntry struct {
|
|
loading bool
|
|
globals starlark.StringDict
|
|
}
|
|
```
|
|
|
|
Key constraints:
|
|
|
|
- **Package-scoped.** Can only load files from within the package's
|
|
own directory. Path traversal (`..`) and absolute paths are rejected.
|
|
One package cannot load another package's files.
|
|
|
|
- **`.star` files only.** Cannot load `.js`, `.json`, or anything
|
|
else. This is a Starlark sandbox, not a file reader.
|
|
|
|
- **Circular dependency detection.** If A loads B loads A → error.
|
|
|
|
- **Cached per invocation.** If `script.star` and `repos.star` both
|
|
load `auth.star`, it executes once. Starlark's thread-level load
|
|
dedup handles this, plus our explicit cache.
|
|
|
|
- **Same predeclared modules.** Loaded files get the same `db`,
|
|
`http`, `json`, `settings`, `connections` modules as the entry
|
|
point. They run in the same permission context.
|
|
|
|
### Starlark Usage
|
|
|
|
```python
|
|
# script.star (entry point)
|
|
load("star/auth.star", "auth_headers", "auth_headers_json")
|
|
load("star/repos.star", "get_repos", "sync_repos")
|
|
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
|
|
load("star/ci.star", "get_ci_status")
|
|
|
|
def on_request(req):
|
|
# ... dispatch to imported functions ...
|
|
|
|
def on_tool_call(tool_name, params):
|
|
# ... dispatch to imported functions ...
|
|
```
|
|
|
|
```python
|
|
# star/auth.star
|
|
def auth_headers(conn):
|
|
return {"Authorization": "token " + conn.get("api_token", "")}
|
|
|
|
def auth_headers_json(conn):
|
|
h = auth_headers(conn)
|
|
h["Content-Type"] = "application/json"
|
|
return h
|
|
```
|
|
|
|
```python
|
|
# star/repos.star
|
|
load("star/auth.star", "auth_headers")
|
|
|
|
def get_repos(conn):
|
|
url = conn["base_url"] + "/api/v1/repos/search?limit=50"
|
|
resp = http.get(url=url, headers=auth_headers(conn))
|
|
if int(resp["status"]) >= 400:
|
|
return None
|
|
return json.decode(resp["body"])
|
|
|
|
def sync_repos(conn):
|
|
repos = get_repos(conn)
|
|
# ... db.insert into private tables ...
|
|
```
|
|
|
|
Submodules can load other submodules. The dependency graph is a DAG
|
|
(cycles are rejected).
|
|
|
|
### Build Script
|
|
|
|
**`packages/build.sh`** already includes `script.star`. Add `star/`:
|
|
|
|
```bash
|
|
local dirs=""
|
|
[ -d "$dir/js" ] && dirs="$dirs js/"
|
|
[ -d "$dir/css" ] && dirs="$dirs css/"
|
|
[ -d "$dir/assets" ] && dirs="$dirs assets/"
|
|
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
|
|
[ -d "$dir/star" ] && dirs="$dirs star/"
|
|
[ -d "$dir/migrations" ] && dirs="$dirs migrations/"
|
|
```
|
|
|
|
### Runner Constructor
|
|
|
|
**`runner.go`** gains `packagesDir`:
|
|
|
|
```go
|
|
type Runner struct {
|
|
sandbox *Sandbox
|
|
stores store.Stores
|
|
packagesDir string // NEW
|
|
notifier NotificationSender
|
|
resolver ProviderResolver
|
|
db *sql.DB
|
|
dbPostgres bool
|
|
}
|
|
|
|
func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
|
|
return &Runner{sandbox: sb, stores: stores}
|
|
}
|
|
|
|
func (r *Runner) SetPackagesDir(dir string) {
|
|
r.packagesDir = dir
|
|
}
|
|
```
|
|
|
|
**`main.go`** wires it:
|
|
|
|
```go
|
|
starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
|
|
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
|
|
```
|
|
|
|
---
|
|
|
|
## Backward Compatibility
|
|
|
|
| Scenario | Behavior |
|
|
|----------|----------|
|
|
| Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. |
|
|
| Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. |
|
|
| New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. |
|
|
| New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. |
|
|
| Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. |
|
|
| Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". |
|
|
|
|
No migration needed. Old packages work as-is. New packages benefit
|
|
from the file-based path. Reinstalling an old package extracts its
|
|
`script.star` to disk (via the updated `extractableRelPath`).
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
| Threat | Mitigation |
|
|
|--------|------------|
|
|
| Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. |
|
|
| Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. |
|
|
| Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. |
|
|
| Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. |
|
|
| Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. |
|
|
| Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. |
|
|
| Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. |
|
|
|
|
---
|
|
|
|
## Changeset Plan
|
|
|
|
| CS | Scope | Files |
|
|
|----|-------|-------|
|
|
| CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. |
|
|
| CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. |
|
|
| CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. |
|
|
| CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. |
|
|
| CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. |
|
|
|
|
CS1-CS4 can land in a single session. CS5 validates. All five are
|
|
backend-only — no FE changes.
|
|
|
|
---
|
|
|
|
## What This Enables
|
|
|
|
- **Library packages** — a `gitea-client` with `script.star` as
|
|
entry point, `star/repos.star`, `star/issues.star`, `star/ci.star`
|
|
as submodules. Clean separation of concerns.
|
|
|
|
- **Large extensions** — a workflow automation package with 20+
|
|
tool handlers doesn't have to be one 2000-line file.
|
|
|
|
- **Shared utilities** — within a single package, common helpers
|
|
(auth, validation, formatting) live in their own file and are
|
|
loaded by multiple entry points.
|
|
|
|
- **Testable modules** — each `.star` file can be loaded and
|
|
tested independently in a dev sandbox.
|
|
|
|
This does NOT enable cross-package loading. Package A cannot load
|
|
Package B's files. That's what `lib.load()` is for (see
|
|
DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is
|
|
intra-package. `lib.load()` is inter-package. Different mechanisms,
|
|
different trust boundaries. |