Feat v0.8.0 files sandbox module
All checks were successful
CI/CD / detect-changes (pull_request) Successful in 4s
CI/CD / test-frontend (pull_request) Has been skipped
CI/CD / test-runners (pull_request) Has been skipped
CI/CD / e2e-smoke (pull_request) Has been skipped
CI/CD / test-go-pg (pull_request) Successful in 2m44s
CI/CD / test-sqlite (pull_request) Successful in 2m57s
CI/CD / build-and-deploy (pull_request) Successful in 1m25s
All checks were successful
CI/CD / detect-changes (pull_request) Successful in 4s
CI/CD / test-frontend (pull_request) Has been skipped
CI/CD / test-runners (pull_request) Has been skipped
CI/CD / e2e-smoke (pull_request) Has been skipped
CI/CD / test-go-pg (pull_request) Successful in 2m44s
CI/CD / test-sqlite (pull_request) Successful in 2m57s
CI/CD / build-and-deploy (pull_request) Successful in 1m25s
Bridge ObjectStore (PVC/S3) into the Starlark sandbox with extension-scoped key namespacing. New files module with put/get/meta/list/delete/exists builtins gated by files.read and files.write permissions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
24
CHANGELOG.md
24
CHANGELOG.md
@@ -2,6 +2,30 @@
|
|||||||
|
|
||||||
All notable changes to Armature are documented here.
|
All notable changes to Armature are documented here.
|
||||||
|
|
||||||
|
## v0.8.0 — Files Module
|
||||||
|
|
||||||
|
New `files` sandbox module. Bridges the existing ObjectStore (PVC/S3) into
|
||||||
|
the Starlark sandbox with extension-scoped key namespacing.
|
||||||
|
|
||||||
|
**files module (permissions: `files.read`, `files.write`)**
|
||||||
|
|
||||||
|
- `files.put(name, content, content_type, metadata)` — store a file with optional metadata companion. Accepts string or bytes content. 50 MB default limit (configurable via `EXT_FILES_MAX_SIZE`).
|
||||||
|
- `files.get(name)` — read a file. Returns dict with `content` (bytes), `content_type`, `size`, `metadata`. Returns None if not found.
|
||||||
|
- `files.meta(name)` — read metadata only (no content transfer).
|
||||||
|
- `files.list(prefix, limit)` — list files by prefix. Returns list of dicts. Filters out internal `_meta/` companions.
|
||||||
|
- `files.delete(name)` — delete a file and its metadata companion. Idempotent.
|
||||||
|
- `files.delete_prefix(prefix)` — delete all files under a prefix.
|
||||||
|
- `files.exists(name)` — check existence without reading.
|
||||||
|
|
||||||
|
**Internal**
|
||||||
|
|
||||||
|
- New file: `sandbox/files_module.go`.
|
||||||
|
- New permission constants: `ExtPermFilesRead`, `ExtPermFilesWrite`.
|
||||||
|
- `ObjectStore` interface gains `List(ctx, prefix, limit)` method; implemented for PVC and S3.
|
||||||
|
- Runner wiring: `SetObjectStore()` setter, `buildModulesWithLibCtx` creates files module when permission granted.
|
||||||
|
- All keys scoped to `ext/{packageID}/`. Metadata stored as companion JSON at `ext/{packageID}/_meta/{name}`.
|
||||||
|
- 16 new tests (15 files module + 1 PVC list).
|
||||||
|
|
||||||
## v0.7.12 — Concurrent Execution Primitive
|
## v0.7.12 — Concurrent Execution Primitive
|
||||||
|
|
||||||
New `batch` sandbox module. Enables extensions to parallelize arbitrary
|
New `batch` sandbox module. Enables extensions to parallelize arbitrary
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Armature — Roadmap
|
# Armature — Roadmap
|
||||||
|
|
||||||
## Current: v0.7.12 — Concurrent Execution Primitive
|
## Current: v0.8.0 — Files Module
|
||||||
|
|
||||||
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
|
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
|
||||||
sandbox, storage, realtime, and ops are kernel primitives. Everything else
|
sandbox, storage, realtime, and ops are kernel primitives. Everything else
|
||||||
@@ -388,14 +388,14 @@ The last major kernel expansion. Completes the primitive set that the
|
|||||||
entire extension ecosystem builds on. After v0.8.x, the kernel is
|
entire extension ecosystem builds on. After v0.8.x, the kernel is
|
||||||
feature-complete for 1.0 — all new capabilities are extensions.
|
feature-complete for 1.0 — all new capabilities are extensions.
|
||||||
|
|
||||||
**v0.8.0 — `files` Module**
|
**v0.8.0 — `files` Module** ✅
|
||||||
|
|
||||||
Bridge `ObjectStore` (PVC/S3) into the Starlark sandbox. Extension-scoped
|
Bridge `ObjectStore` (PVC/S3) into the Starlark sandbox. Extension-scoped
|
||||||
key namespacing (`ext/{packageID}/`). Permissions: `files.read`,
|
key namespacing (`ext/{packageID}/`). Permissions: `files.read`,
|
||||||
`files.write`. Metadata companions stored as sibling objects.
|
`files.write`. Metadata companions stored as sibling objects.
|
||||||
|
|
||||||
New: `sandbox/files_module.go`. Modified: runner wiring, permission
|
New: `sandbox/files_module.go`. Modified: runner wiring, permission
|
||||||
constants.
|
constants. `ObjectStore` interface gained `List()` method. 16 new tests.
|
||||||
|
|
||||||
**v0.8.1 — `workspace` Module**
|
**v0.8.1 — `workspace` Module**
|
||||||
|
|
||||||
|
|||||||
@@ -275,6 +275,48 @@ results, errors = batch.exec([
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### files
|
||||||
|
|
||||||
|
**Permissions:** `files.read`, `files.write`
|
||||||
|
|
||||||
|
Store and retrieve files via the kernel ObjectStore (PVC or S3).
|
||||||
|
All keys are scoped to `ext/{packageID}/` — extensions cannot access
|
||||||
|
each other's files.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Store a file with optional metadata
|
||||||
|
files.put("reports/q1.pdf", pdf_bytes,
|
||||||
|
content_type="application/pdf",
|
||||||
|
metadata={"quarter": "Q1", "year": 2026})
|
||||||
|
|
||||||
|
# Read a file
|
||||||
|
result = files.get("reports/q1.pdf")
|
||||||
|
# result = {"content": b"...", "content_type": "application/pdf",
|
||||||
|
# "size": 12345, "metadata": {"quarter": "Q1", "year": 2026}}
|
||||||
|
|
||||||
|
# Metadata only (no content transfer)
|
||||||
|
meta = files.meta("reports/q1.pdf")
|
||||||
|
|
||||||
|
# List files by prefix
|
||||||
|
entries = files.list(prefix="reports/", limit=50)
|
||||||
|
# entries = [{"name": "reports/q1.pdf", "size": 12345, "content_type": "..."}]
|
||||||
|
|
||||||
|
# Check existence
|
||||||
|
if files.exists("reports/q1.pdf"):
|
||||||
|
files.delete("reports/q1.pdf")
|
||||||
|
|
||||||
|
# Bulk delete
|
||||||
|
files.delete_prefix("temp/")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
- `content` accepts string or bytes; `get()` returns bytes.
|
||||||
|
- Maximum file size: 50 MB (configurable via `EXT_FILES_MAX_SIZE`).
|
||||||
|
- Metadata is stored as a companion JSON object, not in the file itself.
|
||||||
|
- `files.list()` automatically filters out internal metadata companions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Example: automated stage hook
|
## Example: automated stage hook
|
||||||
|
|
||||||
A simple hook that reads a setting, queries data, and advances:
|
A simple hook that reads a setting, queries data, and advances:
|
||||||
|
|||||||
@@ -169,6 +169,9 @@ func main() {
|
|||||||
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
|
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
|
||||||
}
|
}
|
||||||
starlarkRunner.SetBus(bus)
|
starlarkRunner.SetBus(bus)
|
||||||
|
if objStore != nil {
|
||||||
|
starlarkRunner.SetObjectStore(objStore)
|
||||||
|
}
|
||||||
if os.Getenv("EXT_ALLOW_PRIVATE_IPS") == "true" {
|
if os.Getenv("EXT_ALLOW_PRIVATE_IPS") == "true" {
|
||||||
starlarkRunner.SetAllowPrivateIPs(true)
|
starlarkRunner.SetAllowPrivateIPs(true)
|
||||||
log.Printf(" ⚠️ Extension SSRF protection relaxed: private IPs allowed")
|
log.Printf(" ⚠️ Extension SSRF protection relaxed: private IPs allowed")
|
||||||
|
|||||||
@@ -37,6 +37,8 @@ const (
|
|||||||
ExtPermTriggersRegister = "triggers.register"
|
ExtPermTriggersRegister = "triggers.register"
|
||||||
ExtPermRealtimePublish = "realtime.publish"
|
ExtPermRealtimePublish = "realtime.publish"
|
||||||
ExtPermBatchExec = "batch.exec"
|
ExtPermBatchExec = "batch.exec"
|
||||||
|
ExtPermFilesRead = "files.read"
|
||||||
|
ExtPermFilesWrite = "files.write"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ValidExtensionPermissions is the set of recognized permission keys.
|
// ValidExtensionPermissions is the set of recognized permission keys.
|
||||||
@@ -52,6 +54,8 @@ var ValidExtensionPermissions = map[string]bool{
|
|||||||
ExtPermTriggersRegister: true,
|
ExtPermTriggersRegister: true,
|
||||||
ExtPermRealtimePublish: true,
|
ExtPermRealtimePublish: true,
|
||||||
ExtPermBatchExec: true,
|
ExtPermBatchExec: true,
|
||||||
|
ExtPermFilesRead: true,
|
||||||
|
ExtPermFilesWrite: true,
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Extension Permission Model ───────────────
|
// ── Extension Permission Model ───────────────
|
||||||
|
|||||||
472
server/sandbox/files_module.go
Normal file
472
server/sandbox/files_module.go
Normal file
@@ -0,0 +1,472 @@
|
|||||||
|
// Package sandbox — files_module.go
|
||||||
|
//
|
||||||
|
// Permissions: files.read, files.write
|
||||||
|
//
|
||||||
|
// Starlark API:
|
||||||
|
// files.put(name, content, content_type="application/octet-stream", metadata={})
|
||||||
|
// result = files.get(name) → dict or None
|
||||||
|
// meta = files.meta(name) → dict or None
|
||||||
|
// entries = files.list(prefix="", limit=100)
|
||||||
|
// files.delete(name)
|
||||||
|
// files.delete_prefix(prefix)
|
||||||
|
// exists = files.exists(name)
|
||||||
|
//
|
||||||
|
// Bridges the ObjectStore (PVC/S3) into Starlark. All keys are scoped
|
||||||
|
// to ext/{packageID}/ — extensions cannot access each other's files.
|
||||||
|
// Metadata is stored as companion JSON at ext/{packageID}/_meta/{name}.
|
||||||
|
package sandbox
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
|
||||||
|
"go.starlark.net/starlark"
|
||||||
|
"go.starlark.net/starlarkstruct"
|
||||||
|
|
||||||
|
"armature/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
filesDefaultMaxSize = 50 * 1024 * 1024 // 50 MB
|
||||||
|
filesDefaultLimit = 100
|
||||||
|
filesMaxNameLen = 512
|
||||||
|
filesMetaPrefix = "_meta/"
|
||||||
|
)
|
||||||
|
|
||||||
|
// FilesModuleConfig holds the configuration for the files module.
|
||||||
|
type FilesModuleConfig struct {
|
||||||
|
PackageID string
|
||||||
|
CanWrite bool
|
||||||
|
Store storage.ObjectStore
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildFilesModule creates the "files" module.
|
||||||
|
func BuildFilesModule(ctx context.Context, cfg FilesModuleConfig) *starlarkstruct.Module {
|
||||||
|
fns := starlark.StringDict{
|
||||||
|
"get": starlark.NewBuiltin("files.get", filesGet(ctx, cfg)),
|
||||||
|
"meta": starlark.NewBuiltin("files.meta", filesMeta(ctx, cfg)),
|
||||||
|
"list": starlark.NewBuiltin("files.list", filesList(ctx, cfg)),
|
||||||
|
"exists": starlark.NewBuiltin("files.exists", filesExists(ctx, cfg)),
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.CanWrite {
|
||||||
|
fns["put"] = starlark.NewBuiltin("files.put", filesPut(ctx, cfg))
|
||||||
|
fns["delete"] = starlark.NewBuiltin("files.delete", filesDelete(ctx, cfg))
|
||||||
|
fns["delete_prefix"] = starlark.NewBuiltin("files.delete_prefix", filesDeletePrefix(ctx, cfg))
|
||||||
|
}
|
||||||
|
|
||||||
|
return MakeModule("files", fns)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Builtins ──────────────────────────────────
|
||||||
|
|
||||||
|
func filesPut(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var name string
|
||||||
|
var content starlark.Value
|
||||||
|
var contentType string = "application/octet-stream"
|
||||||
|
var metadata *starlark.Dict
|
||||||
|
|
||||||
|
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
|
||||||
|
"name", &name,
|
||||||
|
"content", &content,
|
||||||
|
"content_type?", &contentType,
|
||||||
|
"metadata?", &metadata,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateFileName(name); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.put: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract raw bytes from content (accept String or Bytes).
|
||||||
|
var raw []byte
|
||||||
|
switch v := content.(type) {
|
||||||
|
case starlark.String:
|
||||||
|
raw = []byte(string(v))
|
||||||
|
case starlark.Bytes:
|
||||||
|
raw = []byte(v)
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("files.put: content must be string or bytes, got %s", content.Type())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enforce size limit.
|
||||||
|
maxSize := filesMaxSize()
|
||||||
|
if int64(len(raw)) > maxSize {
|
||||||
|
return nil, fmt.Errorf("files.put: content size %d exceeds limit %d", len(raw), maxSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write main object.
|
||||||
|
key := cfg.scopedKey(name)
|
||||||
|
if err := cfg.Store.Put(ctx, key, bytes.NewReader(raw), int64(len(raw)), contentType); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.put: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write metadata companion if provided.
|
||||||
|
if metadata != nil && metadata.Len() > 0 {
|
||||||
|
metaMap, err := starlarkDictToMap(metadata)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.put: metadata: %w", err)
|
||||||
|
}
|
||||||
|
metaJSON, err := json.Marshal(metaMap)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.put: marshal metadata: %w", err)
|
||||||
|
}
|
||||||
|
metaKey := cfg.metaKey(name)
|
||||||
|
if err := cfg.Store.Put(ctx, metaKey, bytes.NewReader(metaJSON), int64(len(metaJSON)), "application/json"); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.put: write metadata: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return starlark.True, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesGet(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var name string
|
||||||
|
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateFileName(name); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.get: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
key := cfg.scopedKey(name)
|
||||||
|
rc, size, ct, err := cfg.Store.Get(ctx, key)
|
||||||
|
if err == storage.ErrNotFound {
|
||||||
|
return starlark.None, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.get: %w", err)
|
||||||
|
}
|
||||||
|
defer rc.Close()
|
||||||
|
|
||||||
|
data, err := io.ReadAll(rc)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.get: read: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load metadata companion.
|
||||||
|
metaDict := starlark.NewDict(0)
|
||||||
|
metaKey := cfg.metaKey(name)
|
||||||
|
if metaRC, _, _, metaErr := cfg.Store.Get(ctx, metaKey); metaErr == nil {
|
||||||
|
metaBytes, _ := io.ReadAll(metaRC)
|
||||||
|
metaRC.Close()
|
||||||
|
var m map[string]any
|
||||||
|
if json.Unmarshal(metaBytes, &m) == nil {
|
||||||
|
metaDict = mapToStarlarkDict(m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
result := starlark.NewDict(4)
|
||||||
|
result.SetKey(starlark.String("content"), starlark.Bytes(data))
|
||||||
|
result.SetKey(starlark.String("content_type"), starlark.String(ct))
|
||||||
|
result.SetKey(starlark.String("size"), starlark.MakeInt64(size))
|
||||||
|
result.SetKey(starlark.String("metadata"), metaDict)
|
||||||
|
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesMeta(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var name string
|
||||||
|
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateFileName(name); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.meta: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check the main object exists.
|
||||||
|
key := cfg.scopedKey(name)
|
||||||
|
exists, err := cfg.Store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.meta: %w", err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return starlark.None, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load metadata companion.
|
||||||
|
metaDict := starlark.NewDict(0)
|
||||||
|
metaKey := cfg.metaKey(name)
|
||||||
|
if metaRC, _, _, metaErr := cfg.Store.Get(ctx, metaKey); metaErr == nil {
|
||||||
|
metaBytes, _ := io.ReadAll(metaRC)
|
||||||
|
metaRC.Close()
|
||||||
|
var m map[string]any
|
||||||
|
if json.Unmarshal(metaBytes, &m) == nil {
|
||||||
|
metaDict = mapToStarlarkDict(m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return metaDict, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesList(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var prefix string
|
||||||
|
var limit int = filesDefaultLimit
|
||||||
|
|
||||||
|
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
|
||||||
|
"prefix?", &prefix,
|
||||||
|
"limit?", &limit,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if limit <= 0 || limit > 1000 {
|
||||||
|
limit = filesDefaultLimit
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build the full scoped prefix.
|
||||||
|
scopedPrefix := cfg.scopedPrefix()
|
||||||
|
if prefix != "" {
|
||||||
|
// Validate the user-supplied prefix portion.
|
||||||
|
if strings.Contains(prefix, "..") {
|
||||||
|
return nil, fmt.Errorf("files.list: path traversal not allowed")
|
||||||
|
}
|
||||||
|
scopedPrefix += prefix
|
||||||
|
}
|
||||||
|
|
||||||
|
// Request more than limit to account for _meta/ entries we'll filter out.
|
||||||
|
entries, err := cfg.Store.List(ctx, scopedPrefix, limit*2)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.list: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Filter out _meta/ companions and strip the package prefix.
|
||||||
|
pkgPrefix := cfg.scopedPrefix()
|
||||||
|
var results []starlark.Value
|
||||||
|
for _, e := range entries {
|
||||||
|
// Strip package prefix to get the extension-relative key.
|
||||||
|
relKey := strings.TrimPrefix(e.Key, pkgPrefix)
|
||||||
|
|
||||||
|
// Skip metadata companions.
|
||||||
|
if strings.HasPrefix(relKey, filesMetaPrefix) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
d := starlark.NewDict(3)
|
||||||
|
d.SetKey(starlark.String("name"), starlark.String(relKey))
|
||||||
|
d.SetKey(starlark.String("size"), starlark.MakeInt64(e.Size))
|
||||||
|
d.SetKey(starlark.String("content_type"), starlark.String(e.ContentType))
|
||||||
|
results = append(results, d)
|
||||||
|
|
||||||
|
if len(results) >= limit {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return starlark.NewList(results), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesDelete(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var name string
|
||||||
|
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateFileName(name); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.delete: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete main object.
|
||||||
|
key := cfg.scopedKey(name)
|
||||||
|
if err := cfg.Store.Delete(ctx, key); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.delete: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete metadata companion (best-effort, idempotent).
|
||||||
|
metaKey := cfg.metaKey(name)
|
||||||
|
cfg.Store.Delete(ctx, metaKey)
|
||||||
|
|
||||||
|
return starlark.True, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesDeletePrefix(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var prefix string
|
||||||
|
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &prefix); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(prefix, "..") {
|
||||||
|
return nil, fmt.Errorf("files.delete_prefix: path traversal not allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
scopedPrefix := cfg.scopedPrefix() + prefix
|
||||||
|
if err := cfg.Store.DeletePrefix(ctx, scopedPrefix); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.delete_prefix: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also delete metadata companions under this prefix.
|
||||||
|
metaPrefix := cfg.scopedPrefix() + filesMetaPrefix + prefix
|
||||||
|
cfg.Store.DeletePrefix(ctx, metaPrefix)
|
||||||
|
|
||||||
|
return starlark.True, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesExists(ctx context.Context, cfg FilesModuleConfig) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
var name string
|
||||||
|
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateFileName(name); err != nil {
|
||||||
|
return nil, fmt.Errorf("files.exists: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
key := cfg.scopedKey(name)
|
||||||
|
exists, err := cfg.Store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("files.exists: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return starlark.Bool(exists), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Helpers ──────────────────────────────────
|
||||||
|
|
||||||
|
// scopedKey returns the full ObjectStore key for a file name.
|
||||||
|
func (cfg FilesModuleConfig) scopedKey(name string) string {
|
||||||
|
return fmt.Sprintf("ext/%s/%s", cfg.PackageID, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// metaKey returns the ObjectStore key for a file's metadata companion.
|
||||||
|
func (cfg FilesModuleConfig) metaKey(name string) string {
|
||||||
|
return fmt.Sprintf("ext/%s/%s%s", cfg.PackageID, filesMetaPrefix, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// scopedPrefix returns the ObjectStore key prefix for this package.
|
||||||
|
func (cfg FilesModuleConfig) scopedPrefix() string {
|
||||||
|
return fmt.Sprintf("ext/%s/", cfg.PackageID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// validateFileName checks that a file name is safe.
|
||||||
|
func validateFileName(name string) error {
|
||||||
|
if name == "" {
|
||||||
|
return fmt.Errorf("file name is required")
|
||||||
|
}
|
||||||
|
if len(name) > filesMaxNameLen {
|
||||||
|
return fmt.Errorf("file name exceeds %d characters", filesMaxNameLen)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(name, "/") || strings.HasPrefix(name, "\\") {
|
||||||
|
return fmt.Errorf("absolute paths not allowed: %q", name)
|
||||||
|
}
|
||||||
|
if strings.Contains(name, "..") {
|
||||||
|
return fmt.Errorf("path traversal not allowed: %q", name)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(name, filesMetaPrefix) {
|
||||||
|
return fmt.Errorf("names starting with %q are reserved", filesMetaPrefix)
|
||||||
|
}
|
||||||
|
for _, r := range name {
|
||||||
|
if unicode.IsControl(r) {
|
||||||
|
return fmt.Errorf("control characters not allowed in file name")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// filesMaxSize returns the maximum file size from env or default.
|
||||||
|
func filesMaxSize() int64 {
|
||||||
|
if v := os.Getenv("EXT_FILES_MAX_SIZE"); v != "" {
|
||||||
|
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > 0 {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return filesDefaultMaxSize
|
||||||
|
}
|
||||||
|
|
||||||
|
// starlarkDictToMap converts a *starlark.Dict to map[string]any.
|
||||||
|
func starlarkDictToMap(d *starlark.Dict) (map[string]any, error) {
|
||||||
|
m := make(map[string]any, d.Len())
|
||||||
|
for _, item := range d.Items() {
|
||||||
|
k, ok := starlark.AsString(item[0])
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("metadata keys must be strings, got %s", item[0].Type())
|
||||||
|
}
|
||||||
|
m[k] = starlarkValueToGo(item[1])
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// starlarkValueToGo converts a starlark.Value to a Go value for JSON serialization.
|
||||||
|
func starlarkValueToGo(v starlark.Value) any {
|
||||||
|
switch x := v.(type) {
|
||||||
|
case starlark.String:
|
||||||
|
return string(x)
|
||||||
|
case starlark.Int:
|
||||||
|
if i, ok := x.Int64(); ok {
|
||||||
|
return i
|
||||||
|
}
|
||||||
|
return x.String()
|
||||||
|
case starlark.Float:
|
||||||
|
return float64(x)
|
||||||
|
case starlark.Bool:
|
||||||
|
return bool(x)
|
||||||
|
case *starlark.List:
|
||||||
|
out := make([]any, x.Len())
|
||||||
|
for i := 0; i < x.Len(); i++ {
|
||||||
|
out[i] = starlarkValueToGo(x.Index(i))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
case *starlark.Dict:
|
||||||
|
m, _ := starlarkDictToMap(x)
|
||||||
|
return m
|
||||||
|
default:
|
||||||
|
return v.String()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// mapToStarlarkDict converts a map[string]any to *starlark.Dict.
|
||||||
|
func mapToStarlarkDict(m map[string]any) *starlark.Dict {
|
||||||
|
d := starlark.NewDict(len(m))
|
||||||
|
for k, v := range m {
|
||||||
|
d.SetKey(starlark.String(k), goValueToStarlark(v))
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
|
||||||
|
// goValueToStarlark converts a Go value (from JSON) to starlark.Value.
|
||||||
|
func goValueToStarlark(v any) starlark.Value {
|
||||||
|
switch x := v.(type) {
|
||||||
|
case string:
|
||||||
|
return starlark.String(x)
|
||||||
|
case float64:
|
||||||
|
if x == float64(int64(x)) {
|
||||||
|
return starlark.MakeInt64(int64(x))
|
||||||
|
}
|
||||||
|
return starlark.Float(x)
|
||||||
|
case bool:
|
||||||
|
return starlark.Bool(x)
|
||||||
|
case []any:
|
||||||
|
elems := make([]starlark.Value, len(x))
|
||||||
|
for i, e := range x {
|
||||||
|
elems[i] = goValueToStarlark(e)
|
||||||
|
}
|
||||||
|
return starlark.NewList(elems)
|
||||||
|
case map[string]any:
|
||||||
|
return mapToStarlarkDict(x)
|
||||||
|
default:
|
||||||
|
return starlark.None
|
||||||
|
}
|
||||||
|
}
|
||||||
410
server/sandbox/files_module_test.go
Normal file
410
server/sandbox/files_module_test.go
Normal file
@@ -0,0 +1,410 @@
|
|||||||
|
package sandbox
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.starlark.net/starlark"
|
||||||
|
|
||||||
|
"armature/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
// memStore is an in-memory ObjectStore for testing.
|
||||||
|
type memStore struct {
|
||||||
|
objects map[string]memObject
|
||||||
|
}
|
||||||
|
|
||||||
|
type memObject struct {
|
||||||
|
data []byte
|
||||||
|
contentType string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMemStore() *memStore {
|
||||||
|
return &memStore{objects: make(map[string]memObject)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Put(_ context.Context, key string, r io.Reader, size int64, contentType string) error {
|
||||||
|
data, err := io.ReadAll(r)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
m.objects[key] = memObject{data: data, contentType: contentType}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Get(_ context.Context, key string) (io.ReadCloser, int64, string, error) {
|
||||||
|
obj, ok := m.objects[key]
|
||||||
|
if !ok {
|
||||||
|
return nil, 0, "", storage.ErrNotFound
|
||||||
|
}
|
||||||
|
return io.NopCloser(&memReader{data: obj.data}), int64(len(obj.data)), obj.contentType, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Delete(_ context.Context, key string) error {
|
||||||
|
delete(m.objects, key)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) DeletePrefix(_ context.Context, prefix string) error {
|
||||||
|
for k := range m.objects {
|
||||||
|
if strings.HasPrefix(k, prefix) {
|
||||||
|
delete(m.objects, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Exists(_ context.Context, key string) (bool, error) {
|
||||||
|
_, ok := m.objects[key]
|
||||||
|
return ok, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Healthy(_ context.Context) error { return nil }
|
||||||
|
|
||||||
|
func (m *memStore) Stats(_ context.Context) (*storage.StorageStats, error) {
|
||||||
|
return &storage.StorageStats{Backend: "mem", Configured: true, Healthy: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) Backend() string { return "mem" }
|
||||||
|
|
||||||
|
func (m *memStore) List(_ context.Context, prefix string, limit int) ([]storage.ObjectEntry, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
var entries []storage.ObjectEntry
|
||||||
|
for k, obj := range m.objects {
|
||||||
|
if strings.HasPrefix(k, prefix) {
|
||||||
|
entries = append(entries, storage.ObjectEntry{
|
||||||
|
Key: k,
|
||||||
|
Size: int64(len(obj.data)),
|
||||||
|
ContentType: obj.contentType,
|
||||||
|
})
|
||||||
|
if len(entries) >= limit {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// memReader wraps a byte slice as io.ReadCloser.
|
||||||
|
type memReader struct {
|
||||||
|
data []byte
|
||||||
|
pos int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *memReader) Read(p []byte) (int, error) {
|
||||||
|
if r.pos >= len(r.data) {
|
||||||
|
return 0, io.EOF
|
||||||
|
}
|
||||||
|
n := copy(p, r.data[r.pos:])
|
||||||
|
r.pos += n
|
||||||
|
if r.pos >= len(r.data) {
|
||||||
|
return n, io.EOF
|
||||||
|
}
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *memReader) Close() error { return nil }
|
||||||
|
|
||||||
|
// ── Test helpers ──────────────────────────────
|
||||||
|
|
||||||
|
func execWithFiles(t *testing.T, canWrite bool, script string) (*Result, error) {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
store := newMemStore()
|
||||||
|
mod := BuildFilesModule(ctx, FilesModuleConfig{
|
||||||
|
PackageID: "test-pkg",
|
||||||
|
CanWrite: canWrite,
|
||||||
|
Store: store,
|
||||||
|
})
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
return sb.Exec(ctx, "test.star", script, map[string]starlark.Value{
|
||||||
|
"files": mod,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func execWithFilesStore(t *testing.T, store *memStore, canWrite bool, script string) (*Result, error) {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
mod := BuildFilesModule(ctx, FilesModuleConfig{
|
||||||
|
PackageID: "test-pkg",
|
||||||
|
CanWrite: canWrite,
|
||||||
|
Store: store,
|
||||||
|
})
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
return sb.Exec(ctx, "test.star", script, map[string]starlark.Value{
|
||||||
|
"files": mod,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Put + Get round-trip ──────────────────────
|
||||||
|
|
||||||
|
func TestFilesPut_StringContent(t *testing.T) {
|
||||||
|
_, err := execWithFiles(t, true, `
|
||||||
|
files.put("hello.txt", "hello world", content_type="text/plain")
|
||||||
|
result = files.get("hello.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesPut_BytesContent(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("bin.dat", b"\x00\x01\x02\xff")
|
||||||
|
got = files.get("bin.dat")
|
||||||
|
size = got["size"]
|
||||||
|
ct = got["content_type"]
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
size, _ := starlark.AsInt32(result.Globals["size"])
|
||||||
|
if size != 4 {
|
||||||
|
t.Errorf("size = %d, want 4", size)
|
||||||
|
}
|
||||||
|
ct := string(result.Globals["ct"].(starlark.String))
|
||||||
|
if ct != "application/octet-stream" {
|
||||||
|
t.Errorf("content_type = %q, want application/octet-stream", ct)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesPut_WithMetadata(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("doc.pdf", "pdf content", content_type="application/pdf", metadata={"author": "test", "pages": 5})
|
||||||
|
meta = files.meta("doc.pdf")
|
||||||
|
author = meta["author"]
|
||||||
|
pages = meta["pages"]
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
author := string(result.Globals["author"].(starlark.String))
|
||||||
|
if author != "test" {
|
||||||
|
t.Errorf("author = %q, want %q", author, "test")
|
||||||
|
}
|
||||||
|
pages, _ := starlark.AsInt32(result.Globals["pages"])
|
||||||
|
if pages != 5 {
|
||||||
|
t.Errorf("pages = %d, want 5", pages)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesPut_SizeLimit(t *testing.T) {
|
||||||
|
// Set a small limit for testing.
|
||||||
|
t.Setenv("EXT_FILES_MAX_SIZE", "10")
|
||||||
|
_, err := execWithFiles(t, true, `
|
||||||
|
files.put("big.bin", "this content is longer than ten bytes")
|
||||||
|
`)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected size limit error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "exceeds limit") {
|
||||||
|
t.Errorf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Get ───────────────────────────────────────
|
||||||
|
|
||||||
|
func TestFilesGet_NotFound(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, false, `
|
||||||
|
got = files.get("nonexistent.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
if result.Globals["got"] != starlark.None {
|
||||||
|
t.Errorf("expected None, got %v", result.Globals["got"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesGet_ReturnsAllFields(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("test.txt", "data", content_type="text/plain", metadata={"k": "v"})
|
||||||
|
got = files.get("test.txt")
|
||||||
|
has_content = "content" in got
|
||||||
|
has_ct = "content_type" in got
|
||||||
|
has_size = "size" in got
|
||||||
|
has_meta = "metadata" in got
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"has_content", "has_ct", "has_size", "has_meta"} {
|
||||||
|
if result.Globals[key] != starlark.True {
|
||||||
|
t.Errorf("%s = False, want True", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Meta ──────────────────────────────────────
|
||||||
|
|
||||||
|
func TestFilesMeta_NotFound(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, false, `
|
||||||
|
got = files.meta("nonexistent.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
if result.Globals["got"] != starlark.None {
|
||||||
|
t.Errorf("expected None, got %v", result.Globals["got"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesMeta_NoCompanion(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("plain.txt", "data")
|
||||||
|
meta = files.meta("plain.txt")
|
||||||
|
count = len(meta)
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
count, _ := starlark.AsInt32(result.Globals["count"])
|
||||||
|
if count != 0 {
|
||||||
|
t.Errorf("meta length = %d, want 0", count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── List ──────────────────────────────────────
|
||||||
|
|
||||||
|
func TestFilesList_PrefixFilter(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("images/a.png", "a")
|
||||||
|
files.put("images/b.png", "b")
|
||||||
|
files.put("docs/c.pdf", "c")
|
||||||
|
all_entries = files.list()
|
||||||
|
img_entries = files.list(prefix="images/")
|
||||||
|
all_count = len(all_entries)
|
||||||
|
img_count = len(img_entries)
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
allCount, _ := starlark.AsInt32(result.Globals["all_count"])
|
||||||
|
imgCount, _ := starlark.AsInt32(result.Globals["img_count"])
|
||||||
|
|
||||||
|
if allCount != 3 {
|
||||||
|
t.Errorf("all_count = %d, want 3", allCount)
|
||||||
|
}
|
||||||
|
if imgCount != 2 {
|
||||||
|
t.Errorf("img_count = %d, want 2", imgCount)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesList_ExcludesMeta(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("file.txt", "data", metadata={"key": "val"})
|
||||||
|
entries = files.list()
|
||||||
|
count = len(entries)
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
count, _ := starlark.AsInt32(result.Globals["count"])
|
||||||
|
if count != 1 {
|
||||||
|
t.Errorf("count = %d, want 1 (should exclude _meta/ companion)", count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Delete ────────────────────────────────────
|
||||||
|
|
||||||
|
func TestFilesDelete_Idempotent(t *testing.T) {
|
||||||
|
_, err := execWithFiles(t, true, `
|
||||||
|
files.put("temp.txt", "temp")
|
||||||
|
files.delete("temp.txt")
|
||||||
|
files.delete("temp.txt")
|
||||||
|
exists = files.exists("temp.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesDeletePrefix(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("uploads/a.txt", "a")
|
||||||
|
files.put("uploads/b.txt", "b")
|
||||||
|
files.put("keep.txt", "keep")
|
||||||
|
files.delete_prefix("uploads/")
|
||||||
|
a_exists = files.exists("uploads/a.txt")
|
||||||
|
b_exists = files.exists("uploads/b.txt")
|
||||||
|
keep_exists = files.exists("keep.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if result.Globals["a_exists"] != starlark.False {
|
||||||
|
t.Error("uploads/a.txt should be deleted")
|
||||||
|
}
|
||||||
|
if result.Globals["b_exists"] != starlark.False {
|
||||||
|
t.Error("uploads/b.txt should be deleted")
|
||||||
|
}
|
||||||
|
if result.Globals["keep_exists"] != starlark.True {
|
||||||
|
t.Error("keep.txt should be preserved")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Exists ────────────────────────────────────
|
||||||
|
|
||||||
|
func TestFilesExists(t *testing.T) {
|
||||||
|
result, err := execWithFiles(t, true, `
|
||||||
|
files.put("exists.txt", "yes")
|
||||||
|
yes = files.exists("exists.txt")
|
||||||
|
no = files.exists("nope.txt")
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if result.Globals["yes"] != starlark.True {
|
||||||
|
t.Error("exists should be True")
|
||||||
|
}
|
||||||
|
if result.Globals["no"] != starlark.False {
|
||||||
|
t.Error("nope should be False")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Name Validation ──────────────────────────
|
||||||
|
|
||||||
|
func TestFilesNameValidation(t *testing.T) {
|
||||||
|
badNames := []string{
|
||||||
|
"../escape",
|
||||||
|
"/absolute",
|
||||||
|
"_meta/reserved",
|
||||||
|
"",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, name := range badNames {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
script := fmt.Sprintf(`files.exists(%q)`, name)
|
||||||
|
_, err := execWithFiles(t, false, script)
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("expected error for name %q", name)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Write Permission Gate ────────────────────
|
||||||
|
|
||||||
|
func TestFilesWritePermission(t *testing.T) {
|
||||||
|
// Read-only mode: put/delete/delete_prefix should not be available.
|
||||||
|
_, err := execWithFiles(t, false, `
|
||||||
|
files.put("test.txt", "data")
|
||||||
|
`)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error: put should not be available in read-only mode")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "no .put field") && !strings.Contains(err.Error(), "has no .put") {
|
||||||
|
t.Errorf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -33,6 +33,7 @@ import (
|
|||||||
"armature/events"
|
"armature/events"
|
||||||
"armature/metrics"
|
"armature/metrics"
|
||||||
"armature/models"
|
"armature/models"
|
||||||
|
"armature/storage"
|
||||||
"armature/store"
|
"armature/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -77,6 +78,7 @@ type Runner struct {
|
|||||||
dbPostgres bool // true = use $N placeholders; false = use ?
|
dbPostgres bool // true = use $N placeholders; false = use ?
|
||||||
allowPrivateIPs bool
|
allowPrivateIPs bool
|
||||||
bus *events.Bus
|
bus *events.Bus
|
||||||
|
objectStore storage.ObjectStore // nil = files module unavailable
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewRunner creates a runner with the given sandbox and dependencies.
|
// NewRunner creates a runner with the given sandbox and dependencies.
|
||||||
@@ -116,6 +118,11 @@ func (r *Runner) SetBus(bus *events.Bus) {
|
|||||||
r.bus = bus
|
r.bus = bus
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetObjectStore attaches the blob storage backend for the files module.
|
||||||
|
func (r *Runner) SetObjectStore(s storage.ObjectStore) {
|
||||||
|
r.objectStore = s
|
||||||
|
}
|
||||||
|
|
||||||
// SetAllowPrivateIPs disables the SSRF check that blocks connections to
|
// SetAllowPrivateIPs disables the SSRF check that blocks connections to
|
||||||
// private/loopback IPs. For self-hosted environments where extensions
|
// private/loopback IPs. For self-hosted environments where extensions
|
||||||
// reach internal services. Controlled by EXT_ALLOW_PRIVATE_IPS env var.
|
// reach internal services. Controlled by EXT_ALLOW_PRIVATE_IPS env var.
|
||||||
@@ -336,8 +343,9 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
|
|
||||||
modules := make(map[string]starlark.Value)
|
modules := make(map[string]starlark.Value)
|
||||||
|
|
||||||
// Track db permission level: 0=none, 1=read, 2=write
|
// Track tiered permission levels: 0=none, 1=read, 2=write
|
||||||
dbLevel := 0
|
dbLevel := 0
|
||||||
|
filesLevel := 0
|
||||||
hasBatchExec := false
|
hasBatchExec := false
|
||||||
|
|
||||||
for _, perm := range granted {
|
for _, perm := range granted {
|
||||||
@@ -363,6 +371,14 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
case models.ExtPermDBWrite:
|
case models.ExtPermDBWrite:
|
||||||
dbLevel = 2
|
dbLevel = 2
|
||||||
|
|
||||||
|
case models.ExtPermFilesRead:
|
||||||
|
if filesLevel < 1 {
|
||||||
|
filesLevel = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
case models.ExtPermFilesWrite:
|
||||||
|
filesLevel = 2
|
||||||
|
|
||||||
case models.ExtPermWorkflowAccess:
|
case models.ExtPermWorkflowAccess:
|
||||||
modules["workflow"] = BuildWorkflowModule(ctx, r.stores)
|
modules["workflow"] = BuildWorkflowModule(ctx, r.stores)
|
||||||
|
|
||||||
@@ -391,6 +407,15 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Wire files module at the highest granted level.
|
||||||
|
if filesLevel > 0 && r.objectStore != nil {
|
||||||
|
modules["files"] = BuildFilesModule(ctx, FilesModuleConfig{
|
||||||
|
PackageID: packageID,
|
||||||
|
CanWrite: filesLevel == 2,
|
||||||
|
Store: r.objectStore,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// A package reads its own admin + team + user settings.
|
// A package reads its own admin + team + user settings.
|
||||||
userID := ""
|
userID := ""
|
||||||
teamID := ""
|
teamID := ""
|
||||||
|
|||||||
@@ -211,6 +211,78 @@ func (s *PVCStore) Exists(ctx context.Context, key string) (bool, error) {
|
|||||||
return false, fmt.Errorf("storage/pvc: exists %q: %w", key, err)
|
return false, fmt.Errorf("storage/pvc: exists %q: %w", key, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// List returns objects under the given prefix, up to limit entries.
|
||||||
|
func (s *PVCStore) List(ctx context.Context, prefix string, limit int) ([]ObjectEntry, error) {
|
||||||
|
if prefix != "" {
|
||||||
|
if err := validateKey(prefix); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
|
||||||
|
absPrefix := s.resolve(prefix)
|
||||||
|
|
||||||
|
// If the prefix path doesn't exist, return empty.
|
||||||
|
info, err := os.Stat(absPrefix)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("storage/pvc: list %q: %w", prefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var entries []ObjectEntry
|
||||||
|
|
||||||
|
if !info.IsDir() {
|
||||||
|
// Prefix points to a single file — return it.
|
||||||
|
s.mu.RLock()
|
||||||
|
ct := s.mimeMap[prefix]
|
||||||
|
s.mu.RUnlock()
|
||||||
|
if ct == "" {
|
||||||
|
ct = "application/octet-stream"
|
||||||
|
}
|
||||||
|
return []ObjectEntry{{Key: prefix, Size: info.Size(), ContentType: ct}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Walk the directory tree under the prefix.
|
||||||
|
walkRoot := absPrefix
|
||||||
|
err = filepath.Walk(walkRoot, func(path string, fi os.FileInfo, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return nil // skip inaccessible entries
|
||||||
|
}
|
||||||
|
if fi.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if len(entries) >= limit {
|
||||||
|
return filepath.SkipAll
|
||||||
|
}
|
||||||
|
|
||||||
|
// Recover the storage key by stripping basePath + "/".
|
||||||
|
rel, relErr := filepath.Rel(s.basePath, path)
|
||||||
|
if relErr != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
key := filepath.ToSlash(rel)
|
||||||
|
|
||||||
|
s.mu.RLock()
|
||||||
|
ct := s.mimeMap[key]
|
||||||
|
s.mu.RUnlock()
|
||||||
|
if ct == "" {
|
||||||
|
ct = "application/octet-stream"
|
||||||
|
}
|
||||||
|
|
||||||
|
entries = append(entries, ObjectEntry{Key: key, Size: fi.Size(), ContentType: ct})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return entries, fmt.Errorf("storage/pvc: list walk %q: %w", prefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Healthy verifies the storage path is writable.
|
// Healthy verifies the storage path is writable.
|
||||||
func (s *PVCStore) Healthy(ctx context.Context) error {
|
func (s *PVCStore) Healthy(ctx context.Context) error {
|
||||||
probe := filepath.Join(s.basePath, ".health-probe")
|
probe := filepath.Join(s.basePath, ".health-probe")
|
||||||
|
|||||||
@@ -282,6 +282,64 @@ func TestPVC_SubdirCreation(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPVC_List(t *testing.T) {
|
||||||
|
s := tempStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Empty prefix — no files.
|
||||||
|
entries, err := s.List(ctx, "ext/pkg1/", 100)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List empty: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 0 {
|
||||||
|
t.Errorf("expected 0 entries, got %d", len(entries))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add some files under a prefix.
|
||||||
|
for _, name := range []string{"a.txt", "b.png", "sub/c.pdf"} {
|
||||||
|
key := "ext/pkg1/" + name
|
||||||
|
_ = s.Put(ctx, key, bytes.NewReader([]byte("x")), 1, "text/plain")
|
||||||
|
}
|
||||||
|
// Add a file under a different prefix.
|
||||||
|
_ = s.Put(ctx, "ext/pkg2/other.txt", bytes.NewReader([]byte("y")), 1, "text/plain")
|
||||||
|
|
||||||
|
// List all under pkg1.
|
||||||
|
entries, err = s.List(ctx, "ext/pkg1/", 100)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 3 {
|
||||||
|
t.Errorf("expected 3 entries, got %d", len(entries))
|
||||||
|
}
|
||||||
|
|
||||||
|
// List with limit.
|
||||||
|
entries, err = s.List(ctx, "ext/pkg1/", 2)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List limited: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 2 {
|
||||||
|
t.Errorf("expected 2 entries (limit), got %d", len(entries))
|
||||||
|
}
|
||||||
|
|
||||||
|
// List with sub-prefix.
|
||||||
|
entries, err = s.List(ctx, "ext/pkg1/sub/", 100)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List sub: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Errorf("expected 1 entry in sub/, got %d", len(entries))
|
||||||
|
}
|
||||||
|
|
||||||
|
// pkg2 prefix should only find 1.
|
||||||
|
entries, err = s.List(ctx, "ext/pkg2/", 100)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List pkg2: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Errorf("expected 1 entry in pkg2, got %d", len(entries))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestNewPVC_InvalidPath(t *testing.T) {
|
func TestNewPVC_InvalidPath(t *testing.T) {
|
||||||
// Path that can't be created (nested under a file)
|
// Path that can't be created (nested under a file)
|
||||||
tmp := t.TempDir()
|
tmp := t.TempDir()
|
||||||
|
|||||||
@@ -232,6 +232,49 @@ func (s *S3Store) Exists(ctx context.Context, key string) (bool, error) {
|
|||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// List returns objects under the given prefix, up to limit entries.
|
||||||
|
func (s *S3Store) List(ctx context.Context, prefix string, limit int) ([]ObjectEntry, error) {
|
||||||
|
if prefix != "" {
|
||||||
|
if err := validateKey(prefix); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
|
||||||
|
fullPrefix := s.fullKey(prefix)
|
||||||
|
objectsCh := s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{
|
||||||
|
Prefix: fullPrefix,
|
||||||
|
Recursive: true,
|
||||||
|
})
|
||||||
|
|
||||||
|
var entries []ObjectEntry
|
||||||
|
for obj := range objectsCh {
|
||||||
|
if obj.Err != nil {
|
||||||
|
return entries, fmt.Errorf("storage/s3: list %q: %w", prefix, obj.Err)
|
||||||
|
}
|
||||||
|
if len(entries) >= limit {
|
||||||
|
// Drain the remaining channel entries (minio requires it).
|
||||||
|
for range objectsCh {
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// Strip the s3 prefix to recover the storage key.
|
||||||
|
key := strings.TrimPrefix(obj.Key, s.prefix)
|
||||||
|
|
||||||
|
ct := obj.ContentType
|
||||||
|
if ct == "" {
|
||||||
|
ct = "application/octet-stream"
|
||||||
|
}
|
||||||
|
|
||||||
|
entries = append(entries, ObjectEntry{Key: key, Size: obj.Size, ContentType: ct})
|
||||||
|
}
|
||||||
|
|
||||||
|
return entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Healthy checks connectivity by performing a HeadBucket.
|
// Healthy checks connectivity by performing a HeadBucket.
|
||||||
func (s *S3Store) Healthy(ctx context.Context) error {
|
func (s *S3Store) Healthy(ctx context.Context) error {
|
||||||
exists, err := s.client.BucketExists(ctx, s.bucket)
|
exists, err := s.client.BucketExists(ctx, s.bucket)
|
||||||
|
|||||||
@@ -56,10 +56,21 @@ type ObjectStore interface {
|
|||||||
// Stats returns aggregate storage statistics.
|
// Stats returns aggregate storage statistics.
|
||||||
Stats(ctx context.Context) (*StorageStats, error)
|
Stats(ctx context.Context) (*StorageStats, error)
|
||||||
|
|
||||||
|
// List returns objects under the given prefix, up to limit entries.
|
||||||
|
// Returns an empty slice (not error) if no objects match.
|
||||||
|
List(ctx context.Context, prefix string, limit int) ([]ObjectEntry, error)
|
||||||
|
|
||||||
// Backend returns the backend type identifier ("pvc", "s3").
|
// Backend returns the backend type identifier ("pvc", "s3").
|
||||||
Backend() string
|
Backend() string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ObjectEntry represents a single object returned by List.
|
||||||
|
type ObjectEntry struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
ContentType string `json:"content_type"`
|
||||||
|
}
|
||||||
|
|
||||||
// StorageStats holds aggregate storage metrics.
|
// StorageStats holds aggregate storage metrics.
|
||||||
type StorageStats struct {
|
type StorageStats struct {
|
||||||
Backend string `json:"backend"`
|
Backend string `json:"backend"`
|
||||||
|
|||||||
Reference in New Issue
Block a user