# Design: Vector Column Type (v0.8.3)

## Problem

Extensions building semantic search, RAG, or recommendation features need to
store and query high-dimensional vectors (embeddings). Without kernel-level
support, each extension would need to reinvent storage, serialization, and
similarity search — duplicating effort and missing the pgvector optimization
path.

## Design

### Three-tier progressive enhancement

| Backend | Column DDL | Storage | Search |
|---------|-----------|---------|--------|
| Postgres + pgvector | `vector(N)` + HNSW index | Native vector type | `<=>` operator (index-backed) |
| Postgres (no pgvector) | `JSONB` | JSON array | Go-side cosine distance |
| SQLite | `TEXT` | JSON string | Go-side cosine distance |

Extensions declare `"vector(N)"` in their manifest `db_tables` block.
The kernel maps this to the appropriate SQL type at install time based on
detected capabilities.

### Manifest example

```json
{
  "db_tables": {
    "documents": {
      "columns": {
        "title": "text",
        "embedding": "vector(384)"
      }
    }
  },
  "capabilities": {
    "optional": ["pgvector"]
  }
}
```

### API surface

```python
# Insert (vector as list)
db.insert("documents", {"title": "hello", "embedding": [0.1, 0.2, ...]})

# Similarity search
rows = db.query_similar(
    "documents", "embedding",
    vector=[0.1, 0.2, ...],
    limit=10,
    filters={"active": True},
    metric="cosine"
)
# → [{..., "_distance": 0.023}, ...]
```

### Dispatch paths

**pgvector path** — SQL-side computation with index:
```sql
SELECT *, (embedding <=> $1::vector) AS _distance
FROM ext_pkg_documents
WHERE ... ORDER BY embedding <=> $1::vector LIMIT $2
```

**Fallback path** — Go-side computation:
1. `SELECT * FROM table WHERE filters LIMIT 1000`
2. Parse each row's vector column from JSON
3. Compute cosine distance in Go
4. Sort by distance, return top N with `_distance` injected

### Dimension validation

- Manifest: 1 ≤ N ≤ 4096 (validated by `parseVectorDim`)
- Insert-time: no dimension validation (store whatever list is given)
- Query-time: dimension mismatches produce distance = 1.0 (treated as unrelated)

### Performance characteristics

| Path | 1K rows | 10K rows | 100K rows |
|------|---------|----------|-----------|
| pgvector (HNSW) | <1ms | <5ms | <10ms |
| Fallback (Go) | <10ms | ~100ms | Not recommended |

Fallback caps at 1000 rows fetched. Extensions needing large-scale similarity
search should declare `capabilities.optional: ["pgvector"]` and degrade
gracefully.

## Limitations

- Only cosine distance metric (v0.8.3). L2 / inner product can be added later.
- No dimension validation at insert time.
- Fallback path fetches at most 1000 rows — not suitable for large datasets.
- HNSW index is only created for pgvector backends.