Changeset 0.37.16 (#228)
This commit is contained in:
456
docs/DESIGN-MULTI-FILE-STARLARK.md
Normal file
456
docs/DESIGN-MULTI-FILE-STARLARK.md
Normal file
@@ -0,0 +1,456 @@
|
||||
# DESIGN — Multi-File Starlark Packages
|
||||
|
||||
Promotes Starlark scripts from a single inline blob to a proper
|
||||
file tree with `load()` support. Prerequisite for library packages.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
Three problems:
|
||||
|
||||
1. **Scripts live in the DB.** The installer injects `script.star`
|
||||
contents into `manifest["_starlark_script"]`. The runner reads
|
||||
the script from the manifest JSONB column at runtime. This means
|
||||
the entire script — no matter how large — is serialized into a
|
||||
JSON string inside a JSON object inside a database row.
|
||||
|
||||
2. **`load()` is disabled.** The sandbox callback hard-returns an
|
||||
error. There's no way to split code across files.
|
||||
|
||||
3. **Only static assets are extracted.** `extractableRelPath()`
|
||||
allows `js/`, `css/`, `assets/` — nothing else. Starlark files
|
||||
land in the DB, not on disk.
|
||||
|
||||
The hotfix in v0.37.14 (reading `script.star` from the archive and
|
||||
injecting it as `_starlark_script`) made single-file clean but
|
||||
didn't fix the underlying architecture.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
### Archive Structure
|
||||
|
||||
```
|
||||
gitea-client/
|
||||
├── manifest.json ← metadata only, no _starlark_script
|
||||
├── script.star ← entry point (required for starlark tier)
|
||||
├── star/ ← optional: additional modules
|
||||
│ ├── repos.star
|
||||
│ ├── issues.star
|
||||
│ ├── auth.star
|
||||
│ └── ci.star
|
||||
├── js/ ← optional: surface assets
|
||||
│ └── main.js
|
||||
└── css/
|
||||
└── main.css
|
||||
```
|
||||
|
||||
Convention: `script.star` is always the entry point. Submodules live
|
||||
in `star/`. This parallels `js/main.js` as the JS entry point with
|
||||
supporting files alongside it.
|
||||
|
||||
### Manifest Change
|
||||
|
||||
The manifest gains an optional `entry_point` field for packages that
|
||||
want a different entry script name. Default: `script.star`.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "gitea-client",
|
||||
"type": "library",
|
||||
"tier": "starlark",
|
||||
"entry_point": "script.star"
|
||||
}
|
||||
```
|
||||
|
||||
`_starlark_script` is no longer written to the manifest. The field
|
||||
is still read for backward compatibility (existing packages that have
|
||||
it in their DB row continue to work until reinstalled).
|
||||
|
||||
### Extraction
|
||||
|
||||
**`extractableRelPath()`** adds `star/` and bare `*.star` files:
|
||||
|
||||
```go
|
||||
var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}
|
||||
|
||||
func extractableRelPath(name string) string {
|
||||
// Existing prefix matching for directories...
|
||||
|
||||
// Also extract bare .star files at archive root
|
||||
base := filepath.Base(name)
|
||||
if strings.HasSuffix(base, ".star") {
|
||||
// Strip leading directory (package-id/) if present
|
||||
if idx := strings.Index(name, "/"); idx >= 0 {
|
||||
rest := name[idx+1:]
|
||||
if rest == base || strings.HasPrefix(rest, "star/") {
|
||||
return rest
|
||||
}
|
||||
}
|
||||
if name == base {
|
||||
return name
|
||||
}
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
```
|
||||
|
||||
After install, the packages directory looks like:
|
||||
|
||||
```
|
||||
/data/packages/gitea-client/
|
||||
├── script.star
|
||||
├── star/
|
||||
│ ├── repos.star
|
||||
│ ├── issues.star
|
||||
│ ├── auth.star
|
||||
│ └── ci.star
|
||||
├── js/
|
||||
│ └── main.js
|
||||
└── css/
|
||||
└── main.css
|
||||
```
|
||||
|
||||
### Installer Changes
|
||||
|
||||
**`packages.go` `InstallPackage()`:**
|
||||
|
||||
1. Stop injecting `_starlark_script` into the manifest. Remove the
|
||||
hotfix code that reads `script.star` from the archive and stuffs
|
||||
it into the manifest JSON.
|
||||
|
||||
2. Let `extractableRelPath()` handle `.star` files naturally — they
|
||||
get extracted to disk alongside `js/` and `css/`.
|
||||
|
||||
3. Validate: if `tier == "starlark"`, confirm the archive contains
|
||||
`script.star` (or the manifest's `entry_point` value). Return
|
||||
400 if missing.
|
||||
|
||||
### Runner Changes
|
||||
|
||||
**`runner.go` `ExecPackage()`:**
|
||||
|
||||
Replace manifest-based script loading with file-based:
|
||||
|
||||
```go
|
||||
func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
|
||||
if pkg.Status != models.PackageStatusActive {
|
||||
return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
|
||||
}
|
||||
if pkg.Tier != models.ExtTierStarlark {
|
||||
return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
|
||||
}
|
||||
|
||||
// ── Load script from disk (primary) or manifest (legacy) ──
|
||||
script, err := r.loadScript(pkg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
|
||||
}
|
||||
|
||||
// Build package-scoped load callback
|
||||
loader := r.packageLoader(pkg.ID, modules)
|
||||
|
||||
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
|
||||
|
||||
return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
|
||||
}
|
||||
|
||||
func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
|
||||
// Primary: read from disk
|
||||
if r.packagesDir != "" {
|
||||
entryPoint := "script.star"
|
||||
if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
|
||||
entryPoint = ep
|
||||
}
|
||||
path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
|
||||
data, err := os.ReadFile(path)
|
||||
if err == nil && len(data) > 0 {
|
||||
return string(data), nil
|
||||
}
|
||||
// Fall through to legacy
|
||||
}
|
||||
|
||||
// Legacy: inline in manifest
|
||||
script, ok := pkg.Manifest["_starlark_script"].(string)
|
||||
if ok && script != "" {
|
||||
return script, nil
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
|
||||
}
|
||||
```
|
||||
|
||||
### Sandbox Changes
|
||||
|
||||
**`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but
|
||||
accepts a `load` callback:
|
||||
|
||||
```go
|
||||
// LoadFunc resolves load("path") calls to Starlark source.
|
||||
// Returns the module's globals. The sandbox caches results per
|
||||
// thread (Starlark handles this via thread.Load dedup).
|
||||
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)
|
||||
|
||||
func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
|
||||
modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {
|
||||
|
||||
// ... same as Exec() but thread.Load = loader instead of error ...
|
||||
}
|
||||
```
|
||||
|
||||
The existing `Exec()` becomes a thin wrapper that passes a
|
||||
nil/error loader for backward compatibility.
|
||||
|
||||
### Package-Scoped Loader
|
||||
|
||||
**`runner.go`** builds the load callback scoped to the package's
|
||||
directory:
|
||||
|
||||
```go
|
||||
func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
|
||||
if r.packagesDir == "" {
|
||||
return nil // no disk = no load support
|
||||
}
|
||||
|
||||
pkgDir := filepath.Join(r.packagesDir, pkgID)
|
||||
cache := make(map[string]*loadEntry) // dedup + cycle detection
|
||||
|
||||
return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
|
||||
// ── Security: reject traversal ──
|
||||
if strings.Contains(module, "..") || filepath.IsAbs(module) {
|
||||
return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
|
||||
}
|
||||
|
||||
// ── Resolve to package directory ──
|
||||
resolved := filepath.Join(pkgDir, module)
|
||||
if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
|
||||
return nil, fmt.Errorf("load: path escapes package directory: %q", module)
|
||||
}
|
||||
|
||||
// ── Must be a .star file ──
|
||||
if !strings.HasSuffix(resolved, ".star") {
|
||||
return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
|
||||
}
|
||||
|
||||
// ── Cache / cycle detection ──
|
||||
if entry, ok := cache[module]; ok {
|
||||
if entry.loading {
|
||||
return nil, fmt.Errorf("load: circular dependency: %q", module)
|
||||
}
|
||||
return entry.globals, nil
|
||||
}
|
||||
|
||||
entry := &loadEntry{loading: true}
|
||||
cache[module] = entry
|
||||
|
||||
// ── Read and execute ──
|
||||
data, err := os.ReadFile(resolved)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
|
||||
}
|
||||
|
||||
globals, err := starlark.ExecFile(thread, module, string(data),
|
||||
r.predeclaredForLoad(modules))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load: error in %q: %w", module, err)
|
||||
}
|
||||
|
||||
entry.globals = globals
|
||||
entry.loading = false
|
||||
return globals, nil
|
||||
}
|
||||
}
|
||||
|
||||
type loadEntry struct {
|
||||
loading bool
|
||||
globals starlark.StringDict
|
||||
}
|
||||
```
|
||||
|
||||
Key constraints:
|
||||
|
||||
- **Package-scoped.** Can only load files from within the package's
|
||||
own directory. Path traversal (`..`) and absolute paths are rejected.
|
||||
One package cannot load another package's files.
|
||||
|
||||
- **`.star` files only.** Cannot load `.js`, `.json`, or anything
|
||||
else. This is a Starlark sandbox, not a file reader.
|
||||
|
||||
- **Circular dependency detection.** If A loads B loads A → error.
|
||||
|
||||
- **Cached per invocation.** If `script.star` and `repos.star` both
|
||||
load `auth.star`, it executes once. Starlark's thread-level load
|
||||
dedup handles this, plus our explicit cache.
|
||||
|
||||
- **Same predeclared modules.** Loaded files get the same `db`,
|
||||
`http`, `json`, `settings`, `connections` modules as the entry
|
||||
point. They run in the same permission context.
|
||||
|
||||
### Starlark Usage
|
||||
|
||||
```python
|
||||
# script.star (entry point)
|
||||
load("star/auth.star", "auth_headers", "auth_headers_json")
|
||||
load("star/repos.star", "get_repos", "sync_repos")
|
||||
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
|
||||
load("star/ci.star", "get_ci_status")
|
||||
|
||||
def on_request(req):
|
||||
# ... dispatch to imported functions ...
|
||||
|
||||
def on_tool_call(tool_name, params):
|
||||
# ... dispatch to imported functions ...
|
||||
```
|
||||
|
||||
```python
|
||||
# star/auth.star
|
||||
def auth_headers(conn):
|
||||
return {"Authorization": "token " + conn.get("api_token", "")}
|
||||
|
||||
def auth_headers_json(conn):
|
||||
h = auth_headers(conn)
|
||||
h["Content-Type"] = "application/json"
|
||||
return h
|
||||
```
|
||||
|
||||
```python
|
||||
# star/repos.star
|
||||
load("star/auth.star", "auth_headers")
|
||||
|
||||
def get_repos(conn):
|
||||
url = conn["base_url"] + "/api/v1/repos/search?limit=50"
|
||||
resp = http.get(url=url, headers=auth_headers(conn))
|
||||
if int(resp["status"]) >= 400:
|
||||
return None
|
||||
return json.decode(resp["body"])
|
||||
|
||||
def sync_repos(conn):
|
||||
repos = get_repos(conn)
|
||||
# ... db.insert into private tables ...
|
||||
```
|
||||
|
||||
Submodules can load other submodules. The dependency graph is a DAG
|
||||
(cycles are rejected).
|
||||
|
||||
### Build Script
|
||||
|
||||
**`packages/build.sh`** already includes `script.star`. Add `star/`:
|
||||
|
||||
```bash
|
||||
local dirs=""
|
||||
[ -d "$dir/js" ] && dirs="$dirs js/"
|
||||
[ -d "$dir/css" ] && dirs="$dirs css/"
|
||||
[ -d "$dir/assets" ] && dirs="$dirs assets/"
|
||||
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
|
||||
[ -d "$dir/star" ] && dirs="$dirs star/"
|
||||
[ -d "$dir/migrations" ] && dirs="$dirs migrations/"
|
||||
```
|
||||
|
||||
### Runner Constructor
|
||||
|
||||
**`runner.go`** gains `packagesDir`:
|
||||
|
||||
```go
|
||||
type Runner struct {
|
||||
sandbox *Sandbox
|
||||
stores store.Stores
|
||||
packagesDir string // NEW
|
||||
notifier NotificationSender
|
||||
resolver ProviderResolver
|
||||
db *sql.DB
|
||||
dbPostgres bool
|
||||
}
|
||||
|
||||
func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
|
||||
return &Runner{sandbox: sb, stores: stores}
|
||||
}
|
||||
|
||||
func (r *Runner) SetPackagesDir(dir string) {
|
||||
r.packagesDir = dir
|
||||
}
|
||||
```
|
||||
|
||||
**`main.go`** wires it:
|
||||
|
||||
```go
|
||||
starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
|
||||
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
| Scenario | Behavior |
|
||||
|----------|----------|
|
||||
| Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. |
|
||||
| Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. |
|
||||
| New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. |
|
||||
| New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. |
|
||||
| Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. |
|
||||
| Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". |
|
||||
|
||||
No migration needed. Old packages work as-is. New packages benefit
|
||||
from the file-based path. Reinstalling an old package extracts its
|
||||
`script.star` to disk (via the updated `extractableRelPath`).
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
| Threat | Mitigation |
|
||||
|--------|------------|
|
||||
| Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. |
|
||||
| Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. |
|
||||
| Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. |
|
||||
| Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. |
|
||||
| Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. |
|
||||
| Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. |
|
||||
| Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. |
|
||||
|
||||
---
|
||||
|
||||
## Changeset Plan
|
||||
|
||||
| CS | Scope | Files |
|
||||
|----|-------|-------|
|
||||
| CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. |
|
||||
| CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. |
|
||||
| CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. |
|
||||
| CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. |
|
||||
| CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. |
|
||||
|
||||
CS1-CS4 can land in a single session. CS5 validates. All five are
|
||||
backend-only — no FE changes.
|
||||
|
||||
---
|
||||
|
||||
## What This Enables
|
||||
|
||||
- **Library packages** — a `gitea-client` with `script.star` as
|
||||
entry point, `star/repos.star`, `star/issues.star`, `star/ci.star`
|
||||
as submodules. Clean separation of concerns.
|
||||
|
||||
- **Large extensions** — a workflow automation package with 20+
|
||||
tool handlers doesn't have to be one 2000-line file.
|
||||
|
||||
- **Shared utilities** — within a single package, common helpers
|
||||
(auth, validation, formatting) live in their own file and are
|
||||
loaded by multiple entry points.
|
||||
|
||||
- **Testable modules** — each `.star` file can be loaded and
|
||||
tested independently in a dev sandbox.
|
||||
|
||||
This does NOT enable cross-package loading. Package A cannot load
|
||||
Package B's files. That's what `lib.load()` is for (see
|
||||
DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is
|
||||
intra-package. `lib.load()` is inter-package. Different mechanisms,
|
||||
different trust boundaries.
|
||||
Reference in New Issue
Block a user