# asiai — full documentation (English)

Generated: 2026-08-16 19:46 UTC
Pages: 51
Source: https://github.com/druide67/asiai/tree/main/docs
License: Apache-2.0

This file concatenates every English Markdown page so an AI agent can ingest the whole documentation in one fetch. Per-page raw sources are also available individually under https://asiai.dev/markdown/<relative-path>.



---

## /adr/0001-audit-journal-read-for-local-agents

Raw markdown: https://asiai.dev/markdown/adr/0001-audit-journal-read-for-local-agents.md
Rendered: https://asiai.dev/adr/0001-audit-journal-read-for-local-agents/

# ADR 0001 — Audit-journal read access for local agents (MCP)

- Status: accepted
- Date: 2026-07-07
- Scope: `asiai auth login --scope`, `POST /api/v1/fleet/audit-tail`, MCP tool `fleet_audit_tail`

## Context

The fleet audit journal (`~/.local/share/asiai/fleet-audit.jsonl`) records
who did what on the fleet: write commands, their targets, outcomes, source
IPs and token ids. The web dashboard exposes it to a logged-in operator.
Agents (LLM assistants talking to the asiai MCP server) legitimately need
to read it too — "what happened on the fleet tonight?" is a natural
diagnostic question — but the MCP server is a local stdio process with no
authentication of its own, and its output feeds an LLM context that may
leave the machine.

A hard truth shapes the whole design: **no local mechanism can
distinguish the human operator from an agent running under the same user
account.** Any agent with shell access can mint a login code itself.
Biometric approval (Touch ID / LocalAuthentication) would be a real
human-vs-agent barrier but was rejected: headless nodes have no
biometrics, and it is disproportionate for a read. True identity
separation is a future service-account story.

So this gate is **not an access-control barrier against local agents**
— pretending otherwise would be security theater. What it actually buys:

- **Attribution** — every read maps to a code minted at a known time
  from a shell, and the read itself lands in the journal.
- **TTL** — access windows are short (code TTL ≤ 300 s) and single-use;
  nothing persistent accumulates.
- **One auth system** — the same shell-bound operator-code flow the
  dashboard uses, not a second mechanism to audit.

## Decision

Reuse the ephemeral shell-bound operator-login flow, hardened by five
invariants:

1. **Scope is bound to the code at mint, never chosen at exchange —
   and each scope has exactly ONE exchange surface.** `asiai auth login
   --scope audit:read` writes the scope into the code file; every
   consumer inherits it verbatim (`consume_login_code` returns the
   mint-time scope; unknown scopes fail closed). An `audit:read` code
   buys the one-shot redacted exchange and NOTHING else: `/login`
   refuses it (and burns it), so it can never become a session of any
   kind — a 12-hour session on the raw journal route would bypass the
   redaction and bounds this scope exists for. Conversely the one-shot
   exchange refuses (and burns) `full`-scope codes. The raw journal
   endpoint (`GET /api/v1/fleet/audit`, unredacted, for the human
   drawer) requires a full session. Mint decides use.
2. **Redacted output, metadata only.** The exchange response feeds an
   LLM context, so it carries a field whitelist (`ts`, `actor_type`,
   `event`, `source_ip`, `token_id`, `nickname`, `command`, `status`,
   `http_status`, `duration_ms`, `scope`, exchange bookkeeping). Raw
   command arguments (`args`) and free-form `error` text are dropped by
   construction, as is any unknown/future field. No login code, token
   value or secret name ever passes. The whitelisted fields must stay
   closed-set to keep that promise: validation-failure audit lines log
   the submitted `command` only when it is a known command, else a
   fixed `<invalid>` placeholder — free text in a "safe" field would
   smuggle content past the whitelist into the LLM context.
3. **Bounded window + rate limit.** `lines` ≤ 200 (default 50),
   `since_hours` ≤ 24 (default 6), the request body is size-capped
   before parsing (the route is reachable pre-auth), and the route
   charges an all-requests rate limit (6/min per peer) — a leaked
   context cannot exfiltrate the whole history in one sweep, nor
   scrape it in a loop.
4. **REST funnel only.** The MCP tool POSTs to the hub's
   `/api/v1/fleet/audit-tail`; it never opens the journal file or the
   database directly. One read path, one place where redaction and
   journaling happen.
5. **The read is itself journaled** — an `audit_read` event with an
   exchange id and the returned line count — and bounded (the rate
   limit caps the self-referential noise).

The exchange is deliberately **session-less**: one code buys exactly one
redacted read. Managing a cookie/CSRF session lifecycle in a non-browser
consumer would add surface for nothing; with no session there is nothing
to revoke. This is a hardening relative to the session-based design that
was originally reviewed.

## Deployment note — where does the output go?

`fleet_snapshot` / `fleet_health` / `fleet_audit_tail` responses enter
the consuming LLM's context. With a **local** model, the journal never
leaves the machine. With a **cloud** assistant, every (redacted) read
exfiltrates who-did-what metadata to a third party. That trade-off is
the deployer's to make — in a single-operator home lab it is usually
acceptable — but it is a decision, not an accident, and condition 2
exists precisely to keep its blast radius small.

## Alternatives considered

- **Expose the journal unauthenticated on the MCP server** — rejected:
  no attribution, no bound, and a silent default toward exfiltration.
- **A dedicated machine token for reads** — rejected: second credential
  system to rotate/audit, still no human/agent distinction.
- **Touch ID / LocalAuthentication at mint** — rejected (headless nodes,
  disproportionate for reads); revisit with service accounts.
- **Reusing the full web session from the MCP process** — rejected in
  favor of the one-shot exchange (see above).



---

## /adr/0002-community-compare-matching

Raw markdown: https://asiai.dev/markdown/adr/0002-community-compare-matching.md
Rendered: https://asiai.dev/adr/0002-community-compare-matching/

# ADR 0002 — Matching rule for the "This machine vs community" panel

- Status: accepted
- Date: 2026-07-18
- Scope: `GET /api/v1/leaderboard/compare`, Leaderboard page panel

## Context

The Leaderboard page can filter community groups down to the local chip
("This machine"), but it never answers the question users actually come
with: *is my machine in line with everyone else's?* The compare panel
puts local medians (from the local benchmarks DB) next to community
medians (from `api.asiai.dev`), per engine, with a signed delta.

The risky part is not the visual — it is deciding when a local number
and a community group talk about *the same thing*. A wrong match
produces a confidently wrong delta, which is worse than no panel.

## Decision

A local slice and a community group match on the triple
**(chip, model, engine), compared strictly**:

- **Chip** — `collect_hw_chip()` locally vs the group's `hw_chip`,
  case-insensitive string equality after trimming. Both sides come from
  the same asiai collector, so formats agree by construction. No fuzzy
  or substring matching: "Apple M4" must never match "Apple M4 Pro".
  (The community API filters by substring server-side; the strict
  equality re-check happens client-side after the fetch.)
- **Model** — the local model name is passed through
  `normalize_model_name()` — the *same* normalizer used at submission
  time, so what this machine submits is exactly what it later matches
  against — then compared case-insensitively to the group's `model`.
  Quantization is part of the name and therefore part of the identity:
  a Q5_K_XL local run and a Q4_K_M community group do **not** match.
- **Engine** — case-insensitive equality on the display engine name
  (`llamacpp`, `mlx`, `ollama`, …).

Further consequences:

- **Conditions are pooled, not matched.** The v2 aggregates carry one
  median per group; they are not partitioned by ctx/kv-cache/power.
  The panel footer states this honestly ("all submitted conditions
  pooled") instead of implying condition-level matching.
- **Zero match is a first-class state.** When no community group
  survives the strict match, the panel renders the local medians alone
  with an explanatory empty state — it never widens the match to fill
  the void. A "compare against the base model instead" affordance needs
  a base-model notion we do not have; it stays out of v1.
- **Model selection** — the panel compares one model. The route takes
  `?model=`; when the client sends none, the server picks the local
  model with the most runs in the window (deterministic, documented in
  the route). The page seeds it from its model filter, which is a
  **substring** filter (same semantics as the table beside it): the
  server picks the most-benched local model whose normalized name
  contains it. Strict equality governs the *community* match, never
  this local pre-selection — a substring here selects which local
  model to compare, it cannot widen what it compares against.
- **Window** — 30 days on both sides by default (`?days=`, 1–365).
  Same-window comparison only; local 30d vs community 90d would bias
  the delta toward whichever side saw a software upgrade first.

## Alternatives rejected

- **Fuzzy model matching** (prefix/stem match, dropping the quant
  suffix): produces cross-quant deltas that look like hardware or
  engine effects. Rejected — quant is identity.
- **Server-side compare endpoint on api.asiai.dev**: local
  *measurements* never leave the machine; the join happens locally.
  (The community fetch does send the selected model name and chip as
  query parameters — the same metadata a leaderboard filter or a
  submission already sends; the medians themselves stay local.)
- **Matching on chip family with RAM tolerance**: memory bandwidth
  varies within a family; deltas would mix machine classes.

## Consequences

- Fine-tuned or renamed local models will often see zero match — the
  honest outcome, surfaced by the empty state.
- The strict triple makes the panel trivially testable: matching is a
  pure function of two record sets.
- If community groups later become condition-partitioned, the pooled
  footnote and this ADR are the two places to revisit.



---

## /agent

Raw markdown: https://asiai.dev/markdown/agent.md
Rendered: https://asiai.dev/agent/

---
description: Give your AI agents real-time visibility into LLM inference. MCP server with 14 tools for autonomous engine and fleet monitoring.
type: faq
faq:
  - q: "Does asiai require root/sudo?"
    a: "No. GPU observability uses ioreg (no privileges). Power metrics use IOReport which also requires no sudo. The optional --power flag for cross-validation uses sudo powermetrics."
  - q: "What is the API response time?"
    a: "/api/status responds in under 500ms (cached 10s). /api/snapshot takes 1-3s as it collects live data from all engines."
  - q: "Can I run asiai on Linux?"
    a: "No. asiai is macOS Apple Silicon only. It uses sysctl, vm_stat, ioreg, IOReport, and launchd — all macOS-specific APIs."
  - q: "How do I monitor multiple Macs?"
    a: "Run asiai daemon start web --host 0.0.0.0 on each Mac. Scrape /api/metrics with Prometheus. Visualize in Grafana."
  - q: "What if an engine crashes during monitoring?"
    a: "asiai detects engine failures automatically. Use asiai doctor for diagnostics. Set up webhook alerts with asiai alert add for automated notifications."
---

# Agent Integration Guide

> **If you are an AI agent**: use [MCP](#mcp-model-context-protocol) if your framework supports it, otherwise use the [REST API](#rest-api-reference). For setup, see [Quick Start](#quick-start).

## Overview

asiai exposes your Mac's LLM inference infrastructure to AI agents through two mechanisms:

- **MCP Server** — Native tool integration via the [Model Context Protocol](https://modelcontextprotocol.io). Best for AI agents that support MCP (Claude Code, Cursor, Cline, and other MCP-compatible clients).
- **REST API** — Standard HTTP/JSON endpoints. Best for agent frameworks, swarm orchestrators, and any HTTP-capable system (CrewAI, AutoGen, LangGraph, custom agents).

Both give access to the same capabilities:

- **Monitor** system health (CPU, RAM, GPU, thermal, swap)
- **Detect** which inference engines are running and what models are loaded
- **Diagnose** performance issues using GPU observability and inference activity signals
- **Benchmark** models programmatically and track regressions
- **Get recommendations** for the best model/engine based on your hardware

No authentication required for local access. All interfaces bind to `127.0.0.1` by default.

### Which integration should I use?

| Criteria | MCP | REST API |
|----------|-----|----------|
| Your agent supports MCP | **Use MCP** | — |
| Swarm / multi-agent orchestrator | — | **Use REST API** |
| Polling / scheduled monitoring | — | **Use REST API** |
| Prometheus / Grafana integration | — | **Use REST API** |
| Interactive AI assistant (Claude Code, Cursor) | **Use MCP** | — |
| Agent inside Docker container | — | **Use REST API** |
| Custom scripts or automation | — | **Use REST API** |

## Quick Start

### Install asiai

```bash
# Homebrew (recommended)
brew tap druide67/tap && brew install asiai

# pip (with MCP support)
pip install "asiai[mcp]"

# pip (REST API only)
pip install asiai
```

### Option A: MCP Server (for MCP-compatible agents)

```bash
# Start MCP server (stdio transport — used by Claude Code, Cursor, etc.)
asiai mcp
```

No manual server start needed — the MCP client launches `asiai mcp` automatically. See [MCP setup](#mcp-model-context-protocol) below.

### Option B: REST API (for HTTP-based agents)

```bash
# Foreground (development)
asiai web --no-open

# Background daemon (production)
asiai daemon start web
```

The API is available at `http://127.0.0.1:8899`. The port is configurable with `--port`:

```bash
asiai daemon start web --port 8642
```

For remote access (e.g., AI agent on a different machine or from a Docker container):

```bash
asiai daemon start web --host 0.0.0.0
```

> **Note:** If your agent runs inside Docker, `127.0.0.1` is unreachable. Use the host's network IP (e.g., `192.0.2.10`) or `host.docker.internal` on Docker Desktop for Mac.

### Verify

```bash
# REST API
curl http://127.0.0.1:8899/api/status

# MCP (list available tools)
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | asiai mcp
```

---

## MCP (Model Context Protocol)

asiai implements an [MCP server](https://modelcontextprotocol.io) that exposes inference monitoring as native tools. Any MCP-compatible client can connect and use these tools directly — no HTTP setup, no URL management.

### Setup

#### Local (same machine)

Add to your MCP client configuration (e.g., `~/.claude/settings.json` for Claude Code):

```json
{
  "mcpServers": {
    "asiai": {
      "command": "asiai",
      "args": ["mcp"]
    }
  }
}
```

If asiai is installed in a virtualenv:

```json
{
  "mcpServers": {
    "asiai": {
      "command": "/path/to/.venv/bin/asiai",
      "args": ["mcp"]
    }
  }
}
```

#### Remote (different machine via SSH)

```json
{
  "mcpServers": {
    "asiai": {
      "command": "ssh",
      "args": [
        "-o", "ServerAliveInterval=30",
        "-o", "ServerAliveCountMax=3",
        "your-mac-host",
        "cd /path/to/asiai && .venv/bin/asiai mcp"
      ]
    }
  }
}
```

#### SSE transport (network)

For environments that prefer HTTP-based MCP transport:

```bash
asiai mcp --transport sse --host 127.0.0.1 --port 8900
```

### MCP Tools Reference

All tools return JSON. Read-only tools respond in < 2 seconds. `run_benchmark` is the only active operation.

| Tool | Description | Parameters |
|------|-------------|------------|
| `check_inference_health` | Quick health check — engines up/down, memory pressure, thermal, GPU utilization | — |
| `get_inference_snapshot` | Full system state snapshot (stored in SQLite for history) | — |
| `list_models` | All models loaded across all engines with VRAM, quantization, context length | — |
| `detect_engines` | 3-layer detection: config, port scan, process detection. Finds engines on non-standard ports automatically. | — |
| `run_benchmark` | Run benchmark on a model or cross-model comparison. Rate limited: 1 per 60 seconds | `model` (optional), `runs` (1–10, default 3), `compare` (list of strings, optional, mutually exclusive with `model`, max 8) |
| `get_recommendations` | Hardware-aware model/engine recommendations for your chip and RAM | — |
| `diagnose` | Run diagnostic checks (system, engines, daemon health) | — |
| `get_metrics_history` | Historical system metrics from SQLite | `hours` (1–168, default 24) |
| `get_benchmark_history` | Historical benchmark results | `hours` (1–720, default 24), `model` (optional), `engine` (optional) |
| `compare_engines` | Ranked engine comparison with verdict for a given model; supports multi-model comparison from history | `model` (required) |
| `refresh_engines` | Re-detect engines without restarting the MCP server | — |
| `get_fleet_snapshot` | Poll every configured fleet node: reachability, latency, full system + engine snapshots. Uses the same node registry and Bearer tokens as `asiai fleet status` — no web dashboard required | — |
| `get_fleet_health` | Reduced fleet alert feed: unhealthy/degraded engine count + unreachable node count (the dashboard alert-dot reduction) | — |
| `fleet_audit_tail` | Read the fleet audit journal (redacted, metadata only) with a single-use operator code minted via `asiai auth login --scope audit:read`. One code buys exactly one read; see [Fleet audit access for agents](fleet-mode.md#audit-access-for-agents-mcp) | `code` (required), `lines` (1–200, default 50), `since_hours` (≤ 24, default 6) |

### MCP Resources

Static data endpoints, available without calling a tool:

| URI | Description |
|-----|-------------|
| `asiai://status` | Current health status (memory, thermal, GPU) |
| `asiai://models` | All loaded models across engines |
| `asiai://system` | Hardware info (chip, RAM, cores, OS, uptime) |

### MCP Security

- **No sudo**: Power metrics are disabled in MCP mode (`power=False` forced)
- **Rate limiting**: Benchmarks are limited to 1 per 60 seconds
- **Input clamping**: `hours` clamped to 1–168, `runs` clamped to 1–10
- **Local by default**: stdio transport has no network exposure; SSE binds to `127.0.0.1`
- **Audit reads are gated**: `fleet_audit_tail` requires a single-use `audit:read` operator code and returns metadata only (no command payloads, no secrets) — the full design is in [ADR 0001](adr/0001-audit-journal-read-for-local-agents.md)

### MCP Limitations

- **No reconnection**: If the SSH connection drops (network issue, Mac sleep), the MCP server dies and the client must reconnect manually. For unattended monitoring, the REST API with polling is more resilient.
- **Single client**: stdio transport serves one client at a time. Use SSE transport if multiple clients need concurrent access.

---

## REST API Reference

asiai's API is **read-only** — it monitors and reports, but does not control engines. To load/unload models, use engine-native commands (`ollama pull`, `lms load`, etc.).

All endpoints return JSON with HTTP 200. If an engine is unreachable, the response still returns 200 with `"running": false` for that engine — the API itself does not fail.

| Endpoint | Typical response time | Recommended timeout |
|----------|----------------------|---------------------|
| `GET /api/status` | < 500ms (cached 10s) | 2s |
| `GET /api/snapshot` | 1–3s (live collection) | 10s |
| `GET /api/metrics` | < 500ms | 2s |
| `GET /api/history` | < 500ms | 5s |
| `GET /api/engine-history` | < 500ms | 5s |
| `GET /api/benchmarks` | < 500ms | 5s |
| `GET /api/benchmark-process` | < 500ms | 5s |
| `GET /api/bench-runs` | < 500ms | 5s |
| `GET /api/bench-runs/{id}` | < 500ms | 5s |
| `GET /bench/report/{id}.md` | < 500ms | 5s |

### `GET /api/status`

Quick health check. Cached 10 seconds. Response time < 500ms.

**Response:**

```json
{
  "hostname": "mac-mini",
  "chip": "Apple M4 Pro",
  "ram_gb": 64.0,
  "cpu_percent": 12.3,
  "memory_pressure": "normal",
  "gpu_utilization_percent": 45.2,
  "engines": {
    "ollama": {
      "running": true,
      "models_loaded": 2,
      "port": 11434
    },
    "lmstudio": {
      "running": true,
      "models_loaded": 1,
      "port": 1234
    }
  },
  "asiai_version": "1.0.1",
  "uptime_seconds": 86400
}
```

### `GET /api/snapshot`

Full system state. Includes everything from `/api/status` plus detailed model information, GPU metrics, and thermal data.

**Response:**

```json
{
  "system": {
    "hostname": "mac-mini",
    "chip": "Apple M4 Pro",
    "cores_p": 12,
    "cores_e": 4,
    "gpu_cores": 20,
    "ram_total_gb": 64.0,
    "ram_used_gb": 41.2,
    "ram_percent": 64.4,
    "swap_used_gb": 0.0,
    "memory_pressure": "normal",
    "cpu_percent": 12.3,
    "thermal_state": "nominal",
    "gpu_utilization_percent": 45.2,
    "gpu_renderer_percent": 38.1,
    "gpu_tiler_percent": 12.4,
    "gpu_memory_allocated_bytes": 8589934592
  },
  "engines": [
    {
      "name": "ollama",
      "running": true,
      "port": 11434,
      "models": [
        {
          "name": "qwen3.5:latest",
          "size_params": "35B",
          "size_vram_bytes": 21474836480,
          "quantization": "Q4_K_M",
          "context_length": 32768
        }
      ]
    }
  ],
  "timestamp": "2026-03-09T14:30:00Z"
}
```

### `GET /api/metrics`

Prometheus-compatible metrics. Scrape with Prometheus, Datadog, or any compatible tool.

**Response (text/plain):**

```
# HELP asiai_cpu_percent CPU usage percentage
# TYPE asiai_cpu_percent gauge
asiai_cpu_percent 12.3

# HELP asiai_ram_used_gb RAM used in GB
# TYPE asiai_ram_used_gb gauge
asiai_ram_used_gb 41.2

# HELP asiai_gpu_utilization_percent GPU utilization percentage
# TYPE asiai_gpu_utilization_percent gauge
asiai_gpu_utilization_percent 45.2

# HELP asiai_engine_up Engine availability (1=up, 0=down)
# TYPE asiai_engine_up gauge
asiai_engine_up{engine="ollama"} 1
asiai_engine_up{engine="lmstudio"} 1

# HELP asiai_models_loaded Number of models loaded per engine
# TYPE asiai_models_loaded gauge
asiai_models_loaded{engine="ollama"} 2
```

### `GET /api/history?hours=N`

Historical system metrics from SQLite. Default: `hours=24`. Max: `hours=2160` (90 days).

**Response:**

```json
{
  "points": [
    {
      "timestamp": "2026-03-09T14:00:00Z",
      "cpu_percent": 15.2,
      "ram_used_gb": 40.1,
      "ram_percent": 62.7,
      "swap_used_gb": 0.0,
      "memory_pressure": "normal",
      "thermal_state": "nominal",
      "gpu_utilization_percent": 42.0,
      "gpu_renderer_percent": 35.0,
      "gpu_tiler_percent": 10.0,
      "gpu_memory_allocated_bytes": 8589934592
    }
  ],
  "count": 144,
  "hours": 24
}
```

### `GET /api/engine-history?engine=X&hours=N`

Engine-specific activity history. Useful for detecting inference patterns.

**Parameters:**

| Parameter | Required | Default | Description |
|-----------|----------|---------|-------------|
| `engine`  | Yes      | —       | Engine name (ollama, lmstudio, etc.) |
| `hours`   | No       | 24      | Time range |

**Response:**

```json
{
  "engine": "ollama",
  "points": [
    {
      "timestamp": "2026-03-09T14:00:00Z",
      "running": true,
      "tcp_connections": 3,
      "requests_processing": 1,
      "kv_cache_usage_percent": 45.2
    }
  ],
  "count": 144,
  "hours": 24
}
```

### `GET /api/benchmarks?hours=N` · `GET /api/benchmark-process?hours=N`

Benchmark history behind the `/history` charts. `/api/benchmarks` returns
the recorded runs (tok/s, TTFT, power, model, engine, timestamp);
`/api/benchmark-process` returns the engine process metrics sampled during
those runs (CPU %, RSS). Both accept `hours` (and `/api/benchmarks` also
`since`/`until` unix seconds) and return a JSON array, newest first.

## Interpreting Metrics

### `GET /api/bench-runs?type=code&model=&engine=&hours=0&limit=500`

One row per complete bench run of ANY type (`standard`, `agentic`,
`burst`, `code`, `language`, `instruct`, `thinking-ablation`), newest
first, WITHOUT payloads. `score_primary` is only comparable within one
`score_label` — treat the label as part of the value. `gates_failed > 0`
means the run tripped quality gates; do not quote its score without
that caveat.

### `GET /api/bench-runs/{id}`

The same row WITH its full self-describing payload (parsed JSON) — the
exact dict the bench mode produced.

### `GET /bench/report/{id}.md`

The complete markdown report for any persisted run (any type):
labeled headline, metrics with CI95/sample counts, run conditions,
quality gates, provenance. Prefer this over hand-summarizing a payload —
it is the honest, complete rendering.

### System Health Thresholds

| Metric | Normal | Warning | Critical |
|--------|--------|---------|----------|
| `memory_pressure` | `normal` | `warn` | `critical` |
| `ram_percent` | < 75% | 75–90% | > 90% |
| `swap_used_gb` | 0 | 0.1–2.0 | > 2.0 |
| `thermal_state` | `nominal` | `fair` | `serious` / `critical` |
| `cpu_percent` | < 80% | 80–95% | > 95% |

### GPU Thresholds

| Metric | Idle | Active Inference | Overloaded |
|--------|------|------------------|------------|
| `gpu_utilization_percent` | 0–5% | 20–80% | > 90% sustained |
| `gpu_renderer_percent` | 0–5% | 15–70% | > 85% sustained |
| `gpu_memory_allocated_bytes` | < 1 GB | 2–48 GB | > 90% of RAM |

> **Important:** `gpu_utilization_percent = 0` means the GPU is idle, not broken. A value of `-1.0` means the metric is unavailable (e.g., unsupported hardware or collection failure) — do not treat it as "GPU dead".

### Inference Performance

| Metric | Excellent | Good | Degraded |
|--------|-----------|------|----------|
| `tok/s` (7B model) | > 80 | 40–80 | < 40 |
| `tok/s` (35B model) | > 40 | 20–40 | < 20 |
| `tok/s` (70B model) | > 15 | 8–15 | < 8 |
| `TTFT` | < 100ms | 100–500ms | > 500ms |

## Diagnostic Decision Trees

### Slow Generation (low tok/s)

``` mermaid
graph TD
    A["tok/s below expected?"] --> B["Check memory_pressure"]
    A --> C["Check thermal_state"]
    A --> D["Check gpu_utilization_percent"]
    A --> E["Check swap_used_gb"]

    B -->|critical| B1["Models swapping to disk.<br/>Unload models or add RAM."]
    B -->|normal| B2["Continue"]

    C -->|"serious / critical"| C1["Thermal throttling.<br/>Cool down, check airflow."]
    C -->|nominal| C2["Continue"]

    D -->|"< 10%"| D1["GPU not being used.<br/>Check engine config (num_gpu layers)."]
    D -->|"> 90%"| D2["GPU saturated.<br/>Reduce concurrent requests."]
    D -->|"20-80%"| D3["Normal. Check model<br/>quantization and context size."]

    E -->|"> 0"| E1["Model too large for RAM.<br/>Use smaller quantization."]
    E -->|"0"| E2["Check engine version,<br/>try different engine."]
```

### Engine Not Responding

``` mermaid
graph TD
    A["engine.running == false?"] --> B["Check process: lsof -i :port"]
    A --> C["Check memory_pressure"]
    A --> D["Try: asiai doctor"]

    B -->|No process| B1["Engine crashed. Restart it."]
    B -->|Process exists| B2["Engine hung."]

    C -->|critical| C1["OOM killed.<br/>Unload other models first."]
    C -->|normal| C2["Check engine logs."]

    D --> D1["Comprehensive diagnostics"]
```

### High Memory Pressure / VRAM Overflow

``` mermaid
graph TD
    A["memory_pressure == warn/critical?"] --> B["Check swap_used_gb"]
    A --> C["Check models loaded"]
    A --> D["Check gpu_memory_allocated_bytes"]

    B -->|"> 2 GB"| B1["VRAM overflow.<br/>Latency 5-50x worse (disk swap).<br/>Unload models or use Q3_K_S."]
    B -->|"< 2 GB"| B2["Manageable.<br/>Monitor closely."]

    C -->|"Multiple large models"| C1["Unload unused models.<br/>ollama rm / lms unload"]
    C -->|"Single model > 80% RAM"| C2["Use smaller quantization."]

    D --> D1["If > 80% of RAM,<br/>next model load triggers swap."]
```

## Inference Activity Signals

asiai detects active inference through multiple signals:

### GPU Utilization

```
GET /api/snapshot → system.gpu_utilization_percent
```

- **< 5%**: No inference running
- **20–80%**: Active inference (normal range for Apple Silicon unified memory)
- **> 90%**: Heavy inference or multiple concurrent requests

### TCP Connections

```
GET /api/engine-history?engine=ollama&hours=1
```

Each active inference request maintains a TCP connection. A spike in `tcp_connections` indicates active generation.

### Engine-Specific Metrics

For engines that expose `/metrics` (llama.cpp, vllm-mlx):

- `requests_processing > 0`: Active inference
- `kv_cache_usage_percent > 0`: Model has active context

### Correlation Pattern

The most reliable inference detection combines multiple signals:

```python
snapshot = get_snapshot()
gpu_active = snapshot["system"]["gpu_utilization_percent"] > 15
engine_busy = any(
    e.get("tcp_connections", 0) > 0
    for e in snapshot.get("engine_status", [])
)
inference_running = gpu_active and engine_busy
```

## Example Code

### Health Check (Python, stdlib only)

```python
import json
import urllib.request

ASIAI_URL = "http://127.0.0.1:8899"  # Docker: use host IP or host.docker.internal

def check_health():
    """Quick health check. Returns dict with status."""
    req = urllib.request.Request(f"{ASIAI_URL}/api/status")
    with urllib.request.urlopen(req, timeout=5) as resp:
        return json.loads(resp.read())

def is_healthy(status):
    """Interpret health status."""
    issues = []
    if status.get("memory_pressure") != "normal":
        issues.append(f"memory_pressure: {status['memory_pressure']}")
    gpu = status.get("gpu_utilization_percent", 0)
    if gpu > 90:
        issues.append(f"gpu_utilization: {gpu}%")
    engines = status.get("engines", {})
    for name, info in engines.items():
        if not info.get("running"):
            issues.append(f"engine_down: {name}")
    return {"healthy": len(issues) == 0, "issues": issues}

# Usage
status = check_health()
health = is_healthy(status)
if not health["healthy"]:
    print(f"Issues detected: {health['issues']}")
```

### Full System State

```python
def get_full_state():
    """Get complete system snapshot."""
    req = urllib.request.Request(f"{ASIAI_URL}/api/snapshot")
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())

def get_history(hours=24):
    """Get historical metrics."""
    req = urllib.request.Request(f"{ASIAI_URL}/api/history?hours={hours}")
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())

# Detect performance trend
history = get_history(hours=6)
points = history["points"]
if len(points) >= 2:
    recent_gpu = points[-1].get("gpu_utilization_percent", 0)
    earlier_gpu = points[0].get("gpu_utilization_percent", 0)
    if recent_gpu > earlier_gpu * 1.5:
        print("GPU utilization trending up significantly")
```

## Benchmark Cards (Shareable Images)

Generate a shareable benchmark card image with CLI:

```bash
asiai bench --card                    # SVG saved locally (zero dependencies)
asiai bench --card --share            # SVG + PNG via community API
asiai bench --quick --card --share    # Quick bench + card + share (~15s)
```

A **1200x630 dark-themed card** with model, chip, engine comparison bar chart, winner highlight, and metric chips. Optimized for Reddit, X, Discord, and GitHub READMEs.

Cards are saved to `~/.local/share/asiai/cards/` as SVG. Add `--share` to get a PNG download and a shareable URL — PNG is required for posting on Reddit, X, and Discord.

### Via MCP

The `run_benchmark` MCP tool supports card generation with the `card` parameter:

```json
{"tool": "run_benchmark", "arguments": {"model": "qwen3.5", "card": true}}
```

The response includes `card_path` — the absolute path to the SVG file on the MCP server filesystem.

## Webhook Alerts (Push Notifications)

Instead of polling, configure asiai to push notifications when state changes occur:

```bash
# Add a webhook (Slack, Discord, or any URL)
asiai alert add https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# Alerts trigger on:
# - Engine goes down / comes back up
# - Memory pressure transitions (normal → warn → critical)
# - Thermal throttling detected
```

Alerts fire on **transitions only** (not on every check) with a 5-minute cooldown to prevent flooding. This is ideal for swarm orchestrators that need to react to infrastructure changes without continuous polling.

To list or remove alerts: `asiai alert list`, `asiai alert remove <id>`.

## Identify Your Agent

We **recommend** including a `User-Agent` header identifying your agent on REST API requests:

```
User-Agent: asiai-agent/<framework>/<version>
```

Examples:

- `User-Agent: asiai-agent/openclaw/1.2.0`
- `User-Agent: asiai-agent/crewai/0.5`
- `User-Agent: asiai-agent/custom-orchestrator/1.0`

This is **not required** — requests without a User-Agent header work fine. But it helps with:

- **Debugging** — distinguish agent requests from human browser access in logs
- **Per-agent analytics** — see which agents hit the API, how often, and which endpoints
- **Multi-agent environments** — when multiple agents share the same asiai instance, identify who's who

For MCP clients, identification is automatic (the MCP protocol includes client metadata).

---

## Register Your Agent (Optional)

Registration is **entirely optional** — asiai works fully without it. No phone home, no telemetry by default.

If you choose to register, you join the asiai agent network: one API call, no email, no personal data — just your hardware info.

```bash
curl -X POST https://api.asiai.dev/api/v1/agent-register \
  -H "Content-Type: application/json" \
  -d '{
    "chip": "Apple M4 Pro",
    "ram_gb": 64,
    "engines": ["ollama", "lmstudio"],
    "framework": "my-agent",
    "asiai_version": "1.0.1"
  }'
```

**Response:**

```json
{
  "status": "registered",
  "agent_id": "a1b2c3d4e5f6",
  "agent_token": "secret_...",
  "total_agents": 42
}
```

Store `agent_id` and `agent_token` securely in `~/.local/share/asiai/agent.json`. You'll need both for community API calls and benchmark submissions.

### What registration unlocks

| Feature | Without agent_id | With agent_id |
|---------|-----------------|---------------|
| Local bench/monitor/web | Yes | Yes |
| `/api/status`, `/api/snapshot` | Yes | Yes |
| `--share` benchmarks | No | **Yes** |
| `asiai compare` (community) | No | **Yes** |
| `asiai recommend --community` | No | **Yes** |
| Percentile stats | No | **Yes** |
| Agent directory (find peers on same chip) | No | **Yes** |
| Performance alerts (new engine beats yours) | No | **Coming soon** |

### Heartbeat

Keep your registration active with periodic heartbeats:

```bash
curl -X POST https://api.asiai.dev/api/v1/agent-heartbeat \
  -H "Content-Type: application/json" \
  -H "X-Agent-Id: a1b2c3d4e5f6" \
  -H "X-Agent-Token: secret_..." \
  -d '{
    "engines": ["ollama", "lmstudio"],
    "version": "1.0.1",
    "models_loaded": 3,
    "uptime_hours": 72
  }'
```

### Privacy

- **No IP address stored** — your IP is used only for rate limiting and never persisted in the agent registry
- **No personal data** — only hardware info (chip, RAM), engine names, and framework name
- **Opt-in only** — asiai never phones home unless you explicitly register
- **Token security** — your `agent_token` is hashed (SHA-256) before storage; the plaintext is returned only once at registration
- **Rate limit data** — IP hashes (daily-salted SHA-256) in the rate limit table are automatically purged after 30 days

## FAQ

**Q: Does asiai require root/sudo?**
A: No. GPU observability uses `ioreg` (no privileges). Power metrics (`--power` flag in benchmarks) require `sudo powermetrics`, but this is optional.

**Q: What's the API response time?**
A: `/api/status` responds in < 500ms (cached 10s). `/api/snapshot` takes 1–3s (collects live data from all engines).

**Q: Can I run asiai on Linux?**
A: No. asiai is macOS Apple Silicon only. It uses `sysctl`, `vm_stat`, `ioreg`, and `launchd` — all macOS-specific APIs.

**Q: How do I monitor multiple Macs?**
A: Run `asiai daemon start web --host 0.0.0.0` on each Mac. Scrape `/api/metrics` with Prometheus. Visualize in Grafana.

**Q: What if an engine crashes?**
A: asiai detects engine failures automatically. Use `asiai doctor` for diagnostics. Set up webhook alerts with `asiai alert add` for automated notifications.



---

## /agentic-benchmarks

Raw markdown: https://asiai.dev/markdown/agentic-benchmarks.md
Rendered: https://asiai.dev/agentic-benchmarks/

---
description: Agentic-mode benchmark results on Apple Silicon — Qwen3.6 and Qwopus3.6 (27B dense vs 35B-A3B MoE), with and without MTP speculative decoding, across llama.cpp and the MLX engine family. Decode, TTFT, energy, RAM, validity. A living results page.
---

# Agentic Benchmark Results

This page reports real `asiai bench --agentic-mode` results on Apple Silicon. The
agentic protocol runs an 8-phase, prefix-cache-aware conversation (`--runs 5` for
variance), which exercises the way an agent actually uses a model — multi-turn,
long system prefix, 50K-token long-context phase — rather than a single one-shot
generation.

**Why agentic mode — who is this for?** Agent frameworks don't drive a model like a
chatbot: they reuse a large system prefix across many turns, emit tool calls, and
carry long context. A one-shot throughput number misses all of that — and the
ranking can even flip (an engine with great raw decode but a multi-second TTFT or a
broken prefix cache is unusable for an agent). Agentic mode measures the model the
way it is actually driven by **agent orchestrators and coding assistants** — e.g.
[Hermes Agent](https://github.com/nousresearch/hermes-agent),
[OpenClaw](https://github.com/openclaw/openclaw),
[opencode](https://github.com/sst/opencode), Aider, Cline, or Continue — so the
result reflects real agent workloads, not a benchmark artefact.

> **Living document.** These numbers are refreshed as engine versions, model
> revisions and instrumentation improve (e.g. peak-RAM capture). Each row carries
> the exact engine version and model file so a result is always reproducible.

**Campaign 2026-06-03.** Models: Qwen3.6 and the Qwopus3.6 finetune, in two
architectures — **27B dense** and **35B-A3B MoE** (Mixture-of-Experts, ~3B active
parameters per token). Engines: llama.cpp (b9430) and the MLX family (mlx-lm,
mlx_vlm, omlx, rapid-mlx, vllm-mlx). MTP = the model's built-in Multi-Token
Prediction head used for speculative decoding (`--spec-type draft-mtp`).
Hardware: **MacBook Pro M5 Max (128 GB)** and **Mac mini M4 Pro (64 GB)**, both in
High Power Mode.

## How to read the table

Verdict-first. Rows are grouped by a deterministic gate result, not just sorted:

- **★** best validated throughput in the block · **✓** viable · **⚠** reserve
  (passes hard gates but mediocre latency) · **✗** eliminated (failed a gate).
- Gates: `valid ≥ 80%` · `TTFT ≤ 1500 ms` (hard fail > 3000) · `prefix-cache reuse > 0`.
- **dec** = sustained warm decode (tok/s) · **50K** = decode at 50K context ·
  **TTFT** = time-to-first-token (ms) · **t/s/W** = tokens per second per SoC watt
  (efficiency, higher is better) · **RAMpk** = peak engine RSS (GB, the figure that
  governs memory fit) · `—` = not measured (never 0).
- ★ ranks by *throughput only*. Picking a model for real work also weighs output
  quality (see the [dev/code evaluation](dev-quality-benchmarks.md)), which
  throughput does not capture.

> M4 Pro and M5 Max are **not** comparable in absolute terms here — different quant
> (Q5_K_XL vs Q4_K_S). Compare within a machine block.

## MacBook Pro M5 Max 128 GB · Q4

<div class="wide-table" markdown>

| | model · engine · MTP | dec t/s | peak | 50K | TTFT ms | reuse | t/s/W | RAMpk GB | valid% |
|:--|---|--:|--:|--:|--:|--:|--:|--:|--:|
| **★ Tier 1 — winner + fast** |||||||||| |
| ★ | Qwopus-35B · llamacpp b9430 ▲MTP | 123.3 | 127.5 | 83.8 | 67 | 0.8 | 1.590 | — | 100 |
| ✓ | Qwen-35B · llamacpp b9430 ▲MTP | 118.3 | 123.5 | 82.9 | 62 | 0.8 | 1.513 | — | 100 |
| ✓ | Qwopus-35B · llamacpp b9430 | 105.7 | 108.3 | 76.1 | 63 | 0.8 | 1.507 | — | 100 |
| ✓ | Qwen-35B · llamacpp b9430 | 85.5 | 90.8 | 66.7 | 59 | 0.8 | 1.403 | — | 100 |
| **✓ Tier 2 — viable (slower)** |||||||||| |
| ✓ | Qwen-27B · llamacpp b9430 ▲MTP | 28.0 | 29.5 | 22.9 | 118 | 0.8 | 0.378 | 32.2 | 100 |
| ✓ | Qwopus-27B · llamacpp b9430 ▲MTP | 26.7 | 29.8 | 22.0 | 118 | 0.8 | 0.367 | 31.5 | 100 |
| ✓ | Qwopus-27B · llamacpp b9430 | 25.9 | 27.1 | 20.8 | 110 | 0.8 | 0.342 | 28.4 | 100 |
| ✓ | Qwen-27B · llamacpp b9430 | 23.8 | 24.0 | 19.2 | 111 | 0.8 | 0.340 | 28.9 | 100 |
| **⚠ Tier 3 — reserve (poor latency)** |||||||||| |
| ⚠ | Qwopus-27B · mlx-lm 0.31.3 | 29.2 | 29.3 | 24.3 | 600 | 1.0 | 0.461 | 26.4 | 100 |
| ⚠ | Qwen-27B · rapid-mlx 0.6.71 | 20.6 | 20.7 | 17.9 | 798 | — | 0.357 | — | 85 |
| ⚠ | Qwen-27B · omlx 0.4.0 | 20.0 | 20.2 | 17.5 | 2150 | 0.82 | 0.346 | 26.7 | 100 |
| **✗ Tier 4 — eliminated** |||||||||| |
| ✗ | ~~Qwen-27B · mlx_vlm 0.6.0 ▲MTP~~ | ~~41.0~~ | — | — | ~~10879~~ | 0.0 | — | — | 75 |
| ✗ | ~~Qwen-27B · mlx_vlm 0.6.0~~ | ~~31.9~~ | — | 26.0 | ~~9578~~ | 0.0 | — | — | 100 |
| ✗ | ~~Qwen-27B · vllm-mlx 0.3.0~~ | ~~20.5~~ | — | 18.1 | ~~9578~~ | — | — | 24.3 | 100 |

</div>

Eliminations: mlx_vlm+MTP fails validity (75%) and breaks long-context; both
mlx_vlm runs and vllm-mlx have ~9.6 s TTFT (unusable per agent turn).

## Mac mini M4 Pro 64 GB · Q5

<div class="wide-table" markdown>

| | model · engine · MTP | dec t/s | peak | 50K | TTFT ms | reuse | t/s/W | RAMpk GB | valid% |
|:--|---|--:|--:|--:|--:|--:|--:|--:|--:|
| **★ Tier 1** |||||||||| |
| ★ | Qwen-35B · llamacpp b9430 ▲MTP | 44.6 | 50.7 | 32.6 | 143 | 0.8 | 1.557 | 33.0 | 100 |
| ✓ | Qwen-35B · llamacpp b9430 | 36.3 | 45.6 | 29.6 | 133 | 0.8 | 1.553 | 30.8 | 100 |
| **✓ Tier 2** |||||||||| |
| ✓ | Qwen-27B · llamacpp b9430 | 10.4 | 10.4 | 7.2 | 397 | 0.8 | 0.279 | 31.9 | 100 |
| ✓ | Qwen-27B · llamacpp b9430 ▲MTP | 9.7 | 9.8 | 7.5 | 409 | 0.8 | 0.272 | 35.4 | 100 |

</div>

## Key findings

- **The 35B-A3B MoE beats the 27B dense on every throughput axis** on both
  machines — it activates only ~3B parameters per token, so it decodes ~4× faster
  than the dense 27B and is ~3.5× more energy-efficient (1.5 vs ~0.4 tok/s/W).
  Throughput is not quality, however — see the caveat below.
- **Throughput is not agentic fitness.** On an ambiguous-search task — the
  `loop-search` scenario (`asiai bench --instruct`, see
  [dev/code evaluation](dev-quality-benchmarks.md)) — the 35B-A3B **MoE loops
  perfectionistically**: it re-issues semantically-equivalent queries on an
  unresolvable fact until a no-progress guardrail halts it, never producing the
  deliverable. This holds in **both Q4 and Q8** (architectural, not a quant
  artefact), while the **dense 27B never loops**. For an agentic harness such as
  NousResearch's Hermes Agent, this loop-resistance can outweigh the MoE's raw
  decode lead — i.e. the fastest model is not always the right agent.
- **MTP gain depends on architecture × hardware.** Measured decode uplift:
  MoE +38% (M5) / +23% (M4); dense +16% (M5) but **−7% (M4)** — on the slower M4
  GPU the dense draft overhead is not amortised. So MTP is a per-model, per-machine
  measurement, not a universal win.
- **The MLX server family is throughput-only here**: mlx-lm has the best MLX decode
  but a 600 ms TTFT floor; mlx_vlm, vllm-mlx and omlx are knocked out by TTFT
  (2–11 s) and/or broken prefix-cache. llama.cpp dominates first-token latency
  (~60–120 ms).
- **Peak vs steady RAM.** mlx-lm's RSS sits at ~14.5 GB steady but **peaks at
  26.4 GB** (lazy KV allocation + compact MLX-4bit weights); llama.cpp pre-allocates
  the full context KV up front (~29 GB flat). At peak they are comparable — use
  **RAMpk** for memory-fit decisions, not the steady value.

## Methodology & caveats

- `asiai bench --agentic-mode --runs 5`, thinking disabled
  (`chat_template_kwargs.enable_thinking=false`), server context ≥ 65536.
- One engine resident at a time (SOLO); page cache purged between GGUF runs that
  share a file.
- **Quant differs by machine** (M5 Q4_K_S/Q4_K_XL, M4 Q5_K_XL) → absolute numbers
  are not comparable across machines, only within a block.
- **High Power Mode** is required on the M5 laptop (otherwise sustained GPU is
  throttled ~40%); the M4 mini desktop is roughly neutral to it.
- **Known instrumentation gaps** (being fixed): peak RAM is missing (`—`) on some
  manually-launched llama.cpp servers; engine version is not yet stamped per run
  (shown here from a version map); prefix-cache `reuse` is a coarse fraction
  pending a true hit-rate.

See also: [Benchmark methodology](methodology.md) · [Metrics spec](metrics-spec.md)
· [Community leaderboard](leaderboard.md).



---

## /architecture

Raw markdown: https://asiai.dev/markdown/architecture.md
Rendered: https://asiai.dev/architecture/

---
description: How asiai detects engines, collects GPU metrics via IOReport, and stores time-series data. Technical deep-dive.
---

# Architecture

How data flows through asiai — from hardware sensors to your terminal, browser, and AI agents.

## Overview

![asiai architecture overview](assets/architecture.svg)

## Key files

| Layer | Files | Role |
|-------|-------|------|
| **Engines** | `src/asiai/engines/` | ABC `InferenceEngine` + 7 adapters (Ollama, LM Studio, mlx-lm, llama.cpp, oMLX, vllm-mlx, Exo). `OpenAICompatEngine` base class for OpenAI-compatible engines. |
| **Collectors** | `src/asiai/collectors/` | System metrics: `gpu.py` (ioreg), `system.py` (CPU, memory, thermal), `power.py` + `ioreport.py` (GPU/CPU/ANE watts via IOReport), `inference.py` (TCP connections, Prometheus scrape), `snapshot.py` (full system snapshot). |
| **Benchmark** | `src/asiai/benchmark/` | `runner.py` (warmup + N runs, median, stddev, CI95), `prompts.py` (test prompts), `card.py` (SVG card generation). |
| **Storage** | `src/asiai/storage/` | `db.py` (SQLite WAL, all CRUD), `schema.py` (tables + migrations). |
| **CLI** | `src/asiai/cli.py` | Argparse entry point, all 12 commands. |
| **Web** | `src/asiai/web/` | FastAPI + htmx + SSE + ApexCharts dashboard. Routes in `routes/`. |
| **MCP** | `src/asiai/mcp/` | FastMCP server, 11 tools + 3 resources. Transports: stdio, SSE, streamable-http. |
| **Advisor** | `src/asiai/advisor/` | Hardware-aware recommendations (model sizing, engine selection). |
| **Display** | `src/asiai/display/` | ANSI formatters (`formatters.py`), CLI renderer (`cli_renderer.py`), TUI (`tui.py`). |

## Data flow

### Monitoring (daemon mode)

```
Every 60s:
  collectors → snapshot dict → store_snapshot(db) → models table
                                                  → metrics table
  engines    → engine status → store_engine_status(db)
```

### Benchmark

```
CLI --bench → detect engines → pick model → warmup → N runs
           → compute median/stddev/CI95 → store_benchmark(db)
           → render table (ANSI or JSON)
           → optional: --share → POST to community API
           → optional: --card  → generate SVG card
```

### Web dashboard

```
Browser → FastAPI → Jinja2 template (initial render)
       → htmx SSE → /api/v1/stream → real-time updates
       → ApexCharts → /api/v1/metrics?hours=N → historical graphs
```

### MCP server

```
AI agent → stdio/SSE/HTTP → FastMCP → tool call
        → runs collector/benchmark in thread pool (asyncio.to_thread)
        → returns structured JSON
```

## Design principles

1. **Zero dependencies for core** — CLI, collectors, engines, storage use only stdlib Python. Optional extras (`[web]`, `[tui]`, `[mcp]`) add dependencies only when needed.
2. **Shared Data Layer** — The same SQLite database serves CLI, web, MCP, and Prometheus. No separate data stores.
3. **Adapter pattern** — All 10 engines implement `InferenceEngine` ABC. Adding a new engine = 1 file + register in `detect.py`.
4. **Lazy imports** — Each CLI command imports its dependencies locally, keeping startup time fast.
5. **macOS-native** — `ioreg` for GPU, `launchd` for daemons, `lsof` for inference activity. No Linux abstractions.



---

## /bench-modes

Raw markdown: https://asiai.dev/markdown/bench-modes.md
Rendered: https://asiai.dev/bench-modes/

# Benchmark modes

> **Since 1.24**: every mode's complete result is persisted into the
> local database (`bench_runs` table — one row per run, with the full
> payload), charted over time on the web History page, runnable from
> the web Bench page, and exportable as a markdown report with
> `--export FILE.md` (`--export FILE.json` writes the raw payload).

`asiai bench` has three **performance** modes, each answering a different
question, all built on one shared instrumentation brick
(`asiai.benchmark.quality_gates`). The four **quality** modes — `--code`,
`--language`, `--instruct`, `--thinking-ablation` — answer "is the output
*correct*?" rather than "how fast?" and are documented in
[Quality benchmarks](dev-quality-benchmarks.md).

| Mode | Flag | Question it answers |
|------|------|---------------------|
| Standard | *(default)* | How fast / efficient is each engine on a fixed prompt set? (leaderboard) |
| Agentic | `--agentic-mode` | Does the engine reuse a cached system prefix across turns? (multi-turn agents) |
| Burst | `--burst-mode` | How does the engine behave under N concurrent calls? (tool-call fan-out) |

## What each mode *should* capture

Not "what is technically supported" — what is genuinely useful for that mode
versus noise. ✅ capture · ⚠️ capture in a mode-specific shape · ❌ noise / not
applicable.

| Data / metric | Standard | Agentic | Burst |
|---|---|---|---|
| decode tok/s | ✅ per prompt | ✅ per phase | ⚠️ aggregate only (not per-call) |
| TTFT | ✅ | ✅ cold / warm / prefix-hit | ✅ p50/p95/p99 |
| Latency p50/p95/p99/max | ❌ sequential | ❌ sequential | ✅ **the concurrency signal** |
| Aggregate throughput (calls/s, tok/s) | ❌ | ❌ | ✅ **the point of the mode** |
| cached_tokens + prefix-cache verdict | ❌ | ✅ **the point of the mode** | ❌ |
| Multi-run variance | ✅ `--runs` | ✅ `--runs` repeats the protocol (`phase_stats`: median + CV) | ✅ `--burst-runs` |
| **SoC power (watts)** | ✅ per-engine window | ✅ **decode-scoped per-run window** | ✅ **aggregate** over the concurrent window |
| powermetrics cross-validation (sudo) | ✅ leaderboard provenance | ❌ noise (sudo + smears over 2–8 s) | ❌ noise |
| Efficiency tok/s per SoC-watt | ✅ decode | ✅ decode per run | ✅ aggregate throughput |
| **Output validity** (deterministic) | ✅ degenerate gate | ✅ degenerate gate | ✅ arithmetic exact-match |
| thermal_speed_limit | ✅ per run | ✅ summary over 8 phases | ✅ one sample/window |
| thermal **drift** (tok/s slope) | ✅ repeats N identical runs ⇒ slope is meaningful | ❌ phases differ | ❌ no repeated runs |
| early-stop / token-ratio | ✅ | ✅ **catches spec-decode/MTP EOS bugs** | ❌ noise (short answer = correct) |
| duplicate processes | ✅ | ✅ | ✅ |
| memory pressure (swap/swapouts) | ⚠️ pre-check only *(gap, see below)* | ✅ continuous watcher | ✅ continuous watcher (KV blowup under N slots) |

## Three design positions

1. **`soc_watts` means three different things by mode.** Per-engine (standard
   compares engines), decode-scoped per-run window (agentic compares cold vs warm
   — a session average would erase the very signal), aggregate (burst measures
   the energy cost of serving N parallel requests). Hence `read()` vs
   `read_aggregate()` on the probe. `gpu_watts` is kept beside it as a
   diagnostic, but the headline is the full package rail.
2. **powermetrics is only useful for standard.** It is the leaderboard producer,
   where the IOReport↔powermetrics provenance is worth publishing. Elsewhere it
   is sudo friction plus a 500 ms sampler that smears across short windows.
   Hence `cross_validate` is opt-in.
3. **drift and early-stop are noise outside their mode.** Drift only makes sense
   where identical runs repeat (standard); early-stop must not fire on the
   short-but-correct answers of burst.

## Shared instrumentation brick

`asiai.benchmark.quality_gates` is the single source of truth, consumed by all
three performance modes:

- `PowerThermalProbe` — IOReport (no sudo) window sampler.
  - `read()` → `{gpu_watts, soc_watts, energy_joules, thermal_speed_limit, …}`
    (per-window; agentic, burst). `read_power()` is the lighter power-only read
    used to split a window into prefill vs decode at first-token.
  - `read_aggregate()` → provenance dict with `power_source` + `soc_watts` +
    `energy_joules`; powermetrics arm only when `cross_validate=True` (standard
    runner, opt-in).
- `MemoryWatcher` — background swap/swapout watcher (context manager).
- `check_duplicate_processes(engine)` — canonical engine→process pattern map.
- `detect_early_stop(runs)` / `summarize_thermal(runs)`.
- `output_gates.py` — deterministic output validity (degenerate / arithmetic).

`_check_thermal_drift` stays runner-local on purpose: a tok/s slope is only
interpretable across repeated identical runs, which only standard mode produces.

## Metrics generation (1.11.0)

The 1.11.0 audit overhaul changed several formulas; the metrics generation is
tracked by `metrics_version = 3` (standard/leaderboard DB) and
`SCHEMA_VERSION = agentic-v3` (agentic JSON). v3 points must never be aggregated
with older v2/v1 points — the definitions differ:

- **Power headline is SoC, not GPU.** `soc_watts = gpu + cpu + ane + dram + dcs`
  (the DRAM-controller rail). On unified memory a decode is memory-bound, so
  GPU-only badly undercounts (measured idle on M5: GPU 0.07 W vs SoC 21.8 W).
  `gpu_watts` is kept as a diagnostic; the efficiency headline is
  `tok_s_per_soc_watt` (≈ tokens/Joule) with `energy_per_token_j`.
- **Power is decode-scoped** in agentic/burst: the window is rebaselined at
  first-token so watts/energy pair with `decode_tok_s`; prefill is captured
  separately as `prefill_watts`.
- **Token counts are server-exact** (`stream_options.include_usage`,
  `tokens_source='usage'`); the old chars//4 estimate is gone. One unified
  client-side decode formula `(n-1)/(t_last - t_first)` for every engine.
- **Thermal** comes from the notifyd `com.apple.system.thermalpressurelevel`
  channel (the Intel sysctl OID is dead on Apple Silicon).
- **Variance**: `--runs N` repeats the agentic protocol; `phase_stats` reports
  per-phase median + CV. Confidence intervals use the Student-t quantile, not z=2.
- **Output validity**: deterministic gates (`output_gates.py`) flag degenerate
  output; an engine below 80% valid is refused a ranking.
- **Prefix-cache reuse** publishes a raw cross-family signal (`reuse_fraction`,
  `cache_source`, `reuse_corroborated_by_ttft`); the categorical yes/no verdict
  is engine-family-specific and must not be compared across families.

The standard runner now also wraps the continuous `MemoryWatcher` around its
loop (parity with agentic/burst), so swap/swapout growth mid-run is caught, not
just a one-shot pre-check.

## Cross-family campaign protocol

When comparing engines from different families on the *same* model, the engine
must be the only variable that moves. These steps are mandatory, not optional:

1. **Shared GGUF page cache.** llama.cpp, Ollama and LM Studio mmap the *same*
   weights file. The first engine pays the cold disk read; the next finds it
   already warm in the page cache, so its "cold" run is fake. Between GGUF
   engines either `sudo purge` (drop the page cache) or fix the engine order and
   report it — never let an unpurged later engine claim a fast cold load. (MLX
   engines load their own format, so this is GGUF-only.)
2. **Cooldown + clean table between engines.** Unload the previous model, wait
   for memory pressure to return to nominal, and confirm no other inference
   engine is resident (a second engine holding a model competes for GPU and
   memory bandwidth and corrupts both). Run strictly one engine at a time.
3. **Tokenizer trap.** tok/s is only comparable at *comparable token counts*.
   Use `usage.completion_tokens` (server-exact, `tokens_source='usage'`) as
   ground truth, and never compare tok/s across engines whose tokenizers differ
   materially without saying so. With an identical model (same GGUF / same MLX
   repo) the tokenizer is identical and tok/s is directly comparable. Points with
   `tokens_source='chunks'` (the engine didn't report usage) are lower-confidence
   approximations — flag them, don't fold them silently into a cross-family table.
4. **Chat-template concordance.** GGUF and MLX builds can ship different chat
   templates. Assert `prompt_tokens` agrees across engines for the same prompt —
   a divergence means a template mismatch is changing the actual input, which
   invalidates the comparison before it starts.
5. **enable_thinking off, uniformly — and verify it took.** Pass
   `--extra-body '{"chat_template_kwargs":{"enable_thinking":false}}'` so Qwen3
   reasoning tokens don't pollute tok/s/TTFT. The key is engine-specific and is
   *silently ignored* by engines that don't understand it: Ollama's OpenAI-compat
   endpoint wants `{"think": false}` instead. After the run, confirm thinking is
   actually off (no `reasoning_content` in the output) — an engine that ignored
   the flag looks artificially slow and is not comparable.



---

## /benchmark-best-practices

Raw markdown: https://asiai.dev/markdown/benchmark-best-practices.md
Rendered: https://asiai.dev/benchmark-best-practices/

---
description: "How to get accurate LLM benchmark results on Mac: thermal management, background apps, run count, and reproducibility tips."
---

# Benchmark Best Practices

> **Version**: 0.3.2
> **Status**: Living document — updated as methodology evolves
> **References**: MLPerf Inference, SPEC CPU 2017, NVIDIA GenAI-Perf

## Overview

`asiai bench` follows established benchmarking standards to produce **reliable, reproducible,
and comparable** results across inference engines on Apple Silicon. This document tracks
which best practices are implemented, planned, or intentionally excluded.

## Conformance Summary

| Category | Practice | Status | Since |
|----------|----------|--------|-------|
| **Metrics** | TTFT separated from tok/s | Implemented | v0.3.1 |
| | Deterministic sampling (temperature=0) | Implemented | v0.3.2 |
| | Token count from server API (not SSE chunks) | Implemented | v0.3.1 |
| | Per-engine power monitoring | Implemented | v0.3.1 |
| | generation_duration_ms explicit field | Implemented | v0.3.1 |
| **Warmup** | 1 warmup generation per engine (non-timed) | Implemented | v0.3.2 |
| **Runs** | Default 3 runs (SPEC minimum) | Implemented | v0.3.2 |
| | Median as primary metric (SPEC standard) | Implemented | v0.3.2 |
| | Mean + stddev as secondary | Implemented | v0.3.0 |
| **Variance** | Pooled intra-prompt stddev | Implemented | v0.3.1 |
| | CV-based stability classification | Implemented | v0.3.0 |
| **Environment** | Sequential engine execution (memory isolation) | Implemented | v0.1 |
| | Thermal throttling detection + warning | Implemented | v0.3.2 |
| | Thermal level + speed_limit recorded | Implemented | v0.1 |
| **Reproducibility** | Engine version stored per benchmark | Implemented | v0.3.2 |
| | Model format + quantization stored | Implemented | v0.3.2 |
| | Hardware chip + macOS version stored | Implemented | v0.3.2 |
| | Open-source benchmark code | Implemented | v0.1 |
| **Regression** | Historical baseline comparison (SQLite) | Implemented | v0.3.0 |
| | Comparison by (engine, model, prompt_type) | Implemented | v0.3.1 |
| | metrics_version filtering | Implemented | v0.3.1 |
| **Prompts** | 4 diverse prompt types + context fill | Implemented | v0.1 |
| | Fixed max_tokens per prompt | Implemented | v0.1 |

## Planned Improvements

### P1 — Statistical Rigor

| Practice | Description | Standard |
|----------|-------------|----------|
| **95% confidence intervals** | CI = mean +/- 2*SE. More informative than +/- stddev. | Academic |
| **Percentiles (P50/P90/P99)** | For TTFT especially — tail latency matters. | NVIDIA GenAI-Perf |
| **Outlier detection (IQR)** | Flag runs outside [Q1 - 1.5*IQR, Q3 + 1.5*IQR]. | Statistical standard |
| **Trend detection** | Detect monotone performance degradation across runs (thermal drift). | Academic |

### P2 — Reproducibility

| Practice | Description | Standard |
|----------|-------------|----------|
| **Cooldown between engines** | Pause 3-5s between engines to let thermals stabilize. | GPU benchmark |
| **Token ratio verification** | Warn if tokens_generated < 90% of max_tokens. | MLPerf |
| **Export format** | `asiai bench --export` JSON for community submissions. | MLPerf submissions |

### P3 — Advanced

| Practice | Description | Standard |
|----------|-------------|----------|
| **`ignore_eos` option** | Force generation to max_tokens for throughput benchmarks. | NVIDIA |
| **Concurrent request testing** | Test batching throughput (relevant for vllm-mlx). | NVIDIA |
| **Background process audit** | Warn if heavy processes are running during benchmark. | SPEC |

## Intentional Deviations

| Practice | Reason for deviation |
|----------|---------------------|
| **MLPerf minimum 600s duration** | Designed for datacenter GPUs. Local inference on Apple Silicon with 3 runs + 4 prompts already takes ~2-5 minutes. Sufficient for stable results. |
| **SPEC 2 non-timed warmup workloads** | We use 1 warmup generation (not 2 full workloads). Single warmup is sufficient for local inference engines where JIT warmup is minimal. |
| **Population vs sample stddev** | We use population stddev (N divisor) instead of sample stddev (N-1 divisor). With small N (3-5 runs), the difference is minimal and population is more conservative. |
| **Frequency scaling control** | Apple Silicon does not expose CPU governor controls. We record thermal_speed_limit instead to detect throttling. |

## Apple Silicon Specific Considerations

### Unified Memory Architecture

Apple Silicon shares memory between CPU and GPU. Two key implications:

1. **Never benchmark two engines simultaneously** — they compete for the same memory pool.
   `asiai bench` runs engines sequentially by design.
2. **VRAM reporting** — Ollama and LM Studio report `size_vram` natively. For other engines
   (llama.cpp, mlx-lm, oMLX, vLLM-MLX, Exo), asiai uses `ri_phys_footprint` via libproc as a
   fallback estimate. This is what Activity Monitor displays and includes Metal/GPU allocations.
   Estimated values are labeled "(est.)" in the UI.

### Thermal Throttling

- **MacBook Air** (no fan): severe throttling under sustained load. Results degrade after 5-10 min.
- **MacBook Pro** (fan): throttling is mild and usually handled by the fan ramping up.
- **Mac Mini/Studio/Pro**: active cooling, minimal throttling.

`asiai bench` records `thermal_speed_limit` per result and warns if throttling is detected
(speed_limit < 100%) during any run.

### KV Cache and Context Length

Large context sizes (32k+) can cause performance instability on engines that pre-allocate
KV cache at model load time. Example: LM Studio defaults to `loaded_context_length: 262144`
(256k), which allocates ~15-25 GB of KV cache for a 35B model, potentially saturating
64 GB of unified memory.

**Recommendations**:
- When benchmarking large contexts, set engine context length to match the actual test size
  (e.g. `lms load model --context-length 65536` for 64k tests).
- Compare engines with equivalent context length settings for fair results.

## Metadata Stored Per Benchmark

Every benchmark result in SQLite includes:

| Field | Example | Purpose |
|-------|---------|---------|
| `engine` | "ollama" | Engine identification |
| `engine_version` | "0.17.4" | Detect performance changes across updates |
| `model` | "qwen3.5:35b-a3b" | Model identification |
| `model_format` | "gguf" | Differentiate format variants |
| `model_quantization` | "Q4_K_M" | Differentiate quantization levels |
| `hw_chip` | "Apple M4 Pro" | Hardware identification |
| `os_version` | "15.3" | macOS version tracking |
| `thermal_level` | "nominal" | Environment condition |
| `thermal_speed_limit` | 100 | Throttling detection |
| `metrics_version` | 2 | Formula version (prevents cross-version regression) |

This metadata enables:
- **Fair regression comparison**: only compare results with matching metadata
- **Cross-machine benchmarks**: identify hardware differences
- **Community data sharing**: self-describing results (planned for v1.x)



---

## /benchmark-card

Raw markdown: https://asiai.dev/markdown/benchmark-card.md
Rendered: https://asiai.dev/benchmark-card/

---
description: Generate shareable benchmark cards with your results. SVG or PNG, with model, engine, hardware and performance data.
---

# Benchmark Card

Share your benchmark results as a beautiful, branded image. One command generates a card you can post on Reddit, X, Discord, or any social platform.

## Quick start

```bash
asiai bench --quick --card --share    # Bench + card + share in ~15 seconds
asiai bench --card --share            # Full bench + card + share
asiai bench --card                    # SVG + PNG saved locally
```

## Example

![Benchmark card example](assets/benchmark-card-example.png)

## What you get

A **1200x630 dark-themed card** (OG image format, optimized for social media) containing:

- **Hardware badge** — your Apple Silicon chip prominently displayed (top-right)
- **Model name** — which model was benchmarked
- **Engine comparison** — terminal-style bar chart showing tok/s per engine
- **Winner highlight** — which engine is faster and by how much
- **Metric chips** — tok/s, TTFT, stability rating, VRAM usage
- **asiai branding** — logo mark + "asiai.dev" pill badge

The format is designed for maximum readability when shared as a thumbnail on Reddit, X, or Discord.

## How it works

``` mermaid
graph LR
    cmd["asiai bench --card --share"] --> bench["Benchmark<br/>(normal)"]
    bench --> svg["Generate SVG<br/>(zero-dep)"]
    svg --> save["Save local<br/>~/.local/share/asiai/cards/"]
    svg --> share{"--share ?"}
    share -->|Yes| submit["Submit bench<br/>+ get PNG"]
    submit --> url["Shareable URL<br/>+ PNG downloaded"]
```

### Local mode (default)

SVG generated locally with **zero dependencies** — no Pillow, no Cairo, no ImageMagick. Pure Python string templating. Works offline.

Cards are saved to `~/.local/share/asiai/cards/`. SVG is perfect for previewing locally, but **Reddit, X, and Discord require PNG** — add `--share` to get a PNG and a shareable URL.

### Share mode

When combined with `--share`, the benchmark is submitted to the community API, which generates a PNG version server-side. You get:

- A **PNG file** downloaded locally
- A **shareable URL** at `asiai.dev/card/{submission_id}`

## Use cases

### Reddit / r/LocalLLaMA

> "Just benched Qwen 3.5 on my M4 Pro — LM Studio 2.4x faster than Ollama"
> *[attach card image]*

Benchmark posts with images get **5-10x more engagement** than text-only posts.

### X / Twitter

The 1200x630 format is the exact OG image size — it displays perfectly as a card preview in tweets.

### Discord / Slack

Drop the PNG in any channel. The dark theme ensures readability on dark-mode platforms.

### GitHub README

Display your personal benchmark results in your GitHub profile README:

```markdown
![My LLM benchmarks](asiai-card.png)
```

## Combine with --quick

For fast sharing:

```bash
asiai bench -Q --card --share
```

This runs a single prompt (~15 seconds), generates the card, and shares — perfect for quick comparisons after installing a new model or upgrading an engine.

## Design philosophy

Every shared card includes the asiai branding. This creates a **viral loop**:

1. User benchmarks their Mac
2. User shares the card on social media
3. Viewers see the branded card
4. Viewers discover asiai
5. New users benchmark and share their own cards

This is the [Speedtest.net model](https://www.speedtest.net) adapted for local LLM inference.



---

## /benchmark-llm-mac

Raw markdown: https://asiai.dev/markdown/benchmark-llm-mac.md
Rendered: https://asiai.dev/benchmark-llm-mac/

---
title: "How to Benchmark LLMs on Mac"
description: "How to benchmark LLM inference on Mac: step-by-step guide to measure tok/s, TTFT, power, and VRAM on Apple Silicon with multiple engines."
type: howto
date: 2026-03-28
updated: 2026-03-29
duration: PT5M
steps:
  - name: "Install asiai"
    text: "Install asiai via pip (pip install asiai) or Homebrew (brew tap druide67/tap && brew install asiai)."
  - name: "Detect your engines"
    text: "Run 'asiai detect' to automatically find running inference engines (Ollama, LM Studio, llama.cpp, mlx-lm, oMLX, vLLM-MLX, Exo) on your Mac."
  - name: "Run a benchmark"
    text: "Run 'asiai bench' to auto-detect the best model across engines and run a cross-engine comparison measuring tok/s, TTFT, power, and VRAM."
---

# How to Benchmark LLMs on Mac

Running a local LLM on your Mac? Here's how to measure real performance — not vibes, not "it feels fast", but actual tok/s, TTFT, power consumption, and memory usage.

## Why Benchmark?

The same model runs at very different speeds depending on the inference engine. On Apple Silicon, MLX-based engines (LM Studio, mlx-lm, oMLX) can be **2x faster** than llama.cpp-based engines (Ollama) for the same model. Without measuring, you're leaving performance on the table.

## Quick Start (2 minutes)

### 1. Install asiai

```bash
pip install asiai
```

Or via Homebrew:

```bash
brew tap druide67/tap
brew install asiai
```

### 2. Detect your engines

```bash
asiai detect
```

asiai automatically finds running engines (Ollama, LM Studio, llama.cpp, mlx-lm, oMLX, vLLM-MLX, Exo) on your Mac.

### 3. Run a benchmark

```bash
asiai bench
```

That's it. asiai auto-detects the best model across your engines and runs a cross-engine comparison.

## What Gets Measured

| Metric | What It Means |
|--------|--------------|
| **tok/s** | Tokens generated per second (generation only, excludes prompt processing) |
| **TTFT** | Time to First Token — latency before generation starts |
| **Power** | GPU + CPU watts during inference (via IOReport, no sudo needed) |
| **tok/s/W** | Energy efficiency — tokens per second per watt |
| **VRAM** | Memory used by the model (native API or estimated via `ri_phys_footprint`) |
| **Stability** | Run-to-run variance: stable (<5% CV), variable (<10%), unstable (>10%) |
| **Thermal** | Whether your Mac throttled during the benchmark |

## Example Output

```
Mac16,11 — Apple M4 Pro  RAM: 64.0 GB  Pressure: normal

Benchmark: qwen3-coder-30b

  Engine        tok/s   Tokens Duration     TTFT       VRAM    Thermal
  lmstudio      102.2      537    7.00s    0.29s    24.2 GB    nominal
  ollama         69.8      512   17.33s    0.18s    32.0 GB    nominal

  Winner: lmstudio (+46% tok/s)

  Power Efficiency
    lmstudio     102.2 tok/s @ 12.4W = 8.23 tok/s/W
    ollama        69.8 tok/s @ 15.4W = 4.53 tok/s/W
```

*Example output from a real benchmark on M4 Pro 64GB. Your numbers will vary by hardware and model. [See more results →](ollama-vs-lmstudio.md)*

## Advanced Options

### Compare specific engines

```bash
asiai bench --engines ollama,lmstudio,omlx
```

### Multiple prompts and runs

```bash
asiai bench --prompts code,reasoning,tool_call --runs 3
```

### Large context benchmark

```bash
asiai bench --context-size 64K
```

### Generate a shareable card

```bash
asiai bench --card --share
```

Creates a benchmark card image and shares results with the [community leaderboard](leaderboard.md).

## Apple Silicon Tips

### Memory matters

On a 16GB Mac, stick to models under 14GB (loaded). MoE models (Qwen3.5-35B-A3B, 3B active) are ideal — they deliver 35B-class quality at 7B-class memory usage.

### Engine choice matters more than you think

MLX engines are significantly faster than llama.cpp on Apple Silicon for most models. [See our Ollama vs LM Studio comparison](ollama-vs-lmstudio.md) for real numbers.

### Thermal throttling

MacBook Air (no fan) throttles after 5-10 minutes of sustained inference. Mac Mini/Studio/Pro handle sustained workloads without throttling. asiai detects and reports thermal throttling automatically.

## Compare with the Community

See how your Mac stacks up against other Apple Silicon machines:

```bash
asiai compare
```

Or visit the [online leaderboard](leaderboard.md).

## FAQ

**Q: What is the fastest LLM inference engine on Apple Silicon?**
A: In our benchmarks on M4 Pro 64GB, LM Studio (MLX backend) is the fastest for token generation — 46% faster than Ollama (llama.cpp). However, Ollama has lower TTFT (time to first token). See our [detailed comparison](ollama-vs-lmstudio.md).

**Q: How much RAM do I need to run a 30B model on Mac?**
A: A Q4_K_M quantized 30B model uses 24-32 GB of unified memory depending on the engine. You need at least 32 GB RAM, ideally 64 GB to avoid memory pressure. MoE models like Qwen3.5-35B-A3B only use ~7 GB active parameters.

**Q: Does asiai work on Intel Macs?**
A: No. asiai requires Apple Silicon (M1/M2/M3/M4). It uses macOS-specific APIs for GPU metrics, power monitoring, and hardware detection that are only available on Apple Silicon.

**Q: Is Ollama or LM Studio faster on M4?**
A: LM Studio is faster for throughput (102 tok/s vs 70 tok/s on Qwen3-Coder-30B). Ollama is faster for first-token latency (0.18s vs 0.29s) and for large context windows (>32K tokens) where llama.cpp prefill is up to 3x faster.

**Q: How long does a benchmark take?**
A: A quick benchmark takes about 2 minutes. A full cross-engine comparison with multiple prompts and runs takes 10-15 minutes. Use `asiai bench --quick` for a fast single-run test.

**Q: Can I compare my results with other Mac users?**
A: Yes. Run `asiai bench --share` to anonymously submit results to the [community leaderboard](leaderboard.md). Use `asiai compare` to see how your Mac compares to other Apple Silicon machines.

## Further Reading

- [Benchmark Methodology](methodology.md) — how asiai ensures reliable measurements
- [Benchmark Best Practices](benchmark-best-practices.md) — tips for accurate results
- [Engine Comparison](ollama-vs-lmstudio.md) — Ollama vs LM Studio head-to-head



---

## /commands/bench

Raw markdown: https://asiai.dev/markdown/commands/bench.md
Rendered: https://asiai.dev/commands/bench/

---
description: Run side-by-side LLM benchmarks on Apple Silicon. Compare engines, measure tok/s, TTFT, power efficiency. Share results.
---

# asiai bench

Cross-engine benchmark with standardized prompts.

## Usage

```bash
asiai bench [options]
```

## Options

| Option | Description |
|--------|-------------|
| `-m, --model MODEL` | Model to benchmark (default: auto-detect) |
| `-e, --engines LIST` | Filter engines (e.g., `ollama,lmstudio,mlxlm`) |
| `-p, --prompts LIST` | Prompt types: `code`, `tool_call`, `reasoning`, `long_gen` |
| `-r, --runs N` | Runs per prompt (default: 3, for median + stddev) |
| `--power` | Cross-validate power with sudo powermetrics (IOReport always-on) |
| `--context-size SIZE` | Context fill prompt: `4k`, `16k`, `32k`, `64k` |
| `--export FILE` | Export results to JSON file |
| `-H, --history PERIOD` | Show past benchmarks (e.g., `7d`, `24h`) |
| `-Q, --quick` | Quick benchmark: 1 prompt (code), 1 run (~15 seconds) |
| `--compare MODEL [MODEL...]` | Cross-model comparison (2–8 models, mutually exclusive with `-m`) |
| `--card` | Generate a shareable benchmark card (SVG locally, PNG with `--share`) |
| `--share` | Share results to community benchmark database |

## Example

```bash
asiai bench -m qwen3.5 --runs 3 --power
```

```
  Mac Mini M4 Pro — Apple M4 Pro  RAM: 64.0 GB (42% used)  Pressure: normal

Benchmark: qwen3.5

  Engine       tok/s (±stddev)    Tokens   Duration     TTFT       VRAM    Thermal
  ────────── ───────────────── ───────── ────────── ──────── ────────── ──────────
  lmstudio    72.6 ± 0.0 (stable)   435    6.20s    0.28s        —    nominal
  ollama      30.4 ± 0.1 (stable)   448   15.28s    0.25s   26.0 GB   nominal

  Winner: lmstudio (2.4x faster)
  Power: lmstudio 13.2W (5.52 tok/s/W) — ollama 16.0W (1.89 tok/s/W)
```

## Prompts

Four standardized prompts test different generation patterns:

| Name | Tokens | Tests |
|------|--------|-------|
| `code` | 512 | Structured code generation (BST in Python) |
| `tool_call` | 256 | JSON function calling / instruction following |
| `reasoning` | 384 | Multi-step math problem |
| `long_gen` | 1024 | Sustained throughput (bash script) |

Use `--context-size` to test with large context fill prompts instead.

## Cross-engine model matching

The runner resolves model names across engines automatically — `gemma2:9b` (Ollama) and `gemma-2-9b` (LM Studio) are matched as the same model.

## JSON export

Export results for sharing or analysis:

```bash
asiai bench -m qwen3.5 --export bench.json
```

The JSON includes machine metadata, per-engine statistics (median, CI 95%, P50/P90/P99), raw per-run data, and a schema version for forward compatibility.

## Regression detection

After each benchmark, asiai compares results against the last 7 days of history and warns about performance regressions (e.g., after an engine update or macOS upgrade).

## Quick benchmark

Run a fast benchmark with a single prompt and one run (~15 seconds):

```bash
asiai bench --quick
asiai bench -Q -m qwen3.5
```

This is ideal for demos, GIFs, and quick checks. The `code` prompt is used by default. You can override with `--prompts` if needed.

## Cross-model comparison

Compare multiple models in a single session with `--compare`:

```bash
# Auto-expand across all available engines
asiai bench --compare qwen3.5:4b deepseek-r1:7b

# Filter to a specific engine
asiai bench --compare qwen3.5:4b deepseek-r1:7b -e ollama

# Pin each model to an engine with @
asiai bench --compare qwen3.5:4b@lmstudio deepseek-r1:7b@ollama
```

The `@` notation splits on the **last** `@` in the string, so model names containing `@` are handled correctly.

### Rules

- `--compare` and `--model` are **mutually exclusive** — use one or the other.
- Accepts 2 to 8 model slots.
- Without `@`, each model is expanded to every engine where it is available.

### Session types

The session type is detected automatically based on the slot list:

| Type | Condition | Example |
|------|-----------|---------|
| **engine** | Same model, different engines | `--compare qwen3.5:4b@lmstudio qwen3.5:4b@ollama` |
| **model** | Different models, same engine | `--compare qwen3.5:4b deepseek-r1:7b -e ollama` |
| **matrix** | Mixed models and engines | `--compare qwen3.5:4b@lmstudio deepseek-r1:7b@ollama` |

### Combined with other flags

`--compare` works with all output and run flags:

```bash
asiai bench --compare qwen3.5:4b deepseek-r1:7b --quick
asiai bench --compare qwen3.5:4b deepseek-r1:7b --card --share
asiai bench --compare qwen3.5:4b deepseek-r1:7b --runs 5 --power
```

## Benchmark card

Generate a shareable benchmark card:

```bash
asiai bench --card                    # SVG saved locally
asiai bench --card --share            # SVG + PNG (via community API)
asiai bench --quick --card --share    # Quick bench + card + share
```

The card is a 1200x630 dark-themed image with:
- Model name and hardware chip badge
- Specs banner: quantization, RAM, GPU cores, context size
- Terminal-style bar chart of tok/s per engine
- Winner highlight with delta (e.g., "2.4x")
- Metric chips: tok/s, TTFT, stability, VRAM, power (W + tok/s/W), engine version
- asiai branding

The SVG is saved to `~/.local/share/asiai/cards/`. With `--share`, a PNG is also downloaded from the API.

## Community sharing

Share your results anonymously:

```bash
asiai bench --share
```

View the community leaderboard with `asiai leaderboard`.

## Thermal drift detection

When running 3+ runs, asiai detects monotone tok/s degradation across consecutive runs. If tok/s drops consistently (>5%), a warning is emitted indicating possible thermal throttling buildup.



---

## /commands/compare

Raw markdown: https://asiai.dev/markdown/commands/compare.md
Rendered: https://asiai.dev/commands/compare/

---
description: Cross-model and cross-engine benchmark matrix. Compare up to 8 model@engine combinations in a single run.
---

# asiai compare

Compare your local benchmarks against community data.

## Usage

```bash
asiai compare [options]
```

## Options

| Option | Description |
|--------|-------------|
| `--chip CHIP` | Apple Silicon chip to compare against (default: auto-detect) |
| `--model MODEL` | Filter by model name |
| `--db PATH` | Path to local benchmark database |

## Example

```bash
asiai compare --model qwen3.5
```

```
  Compare: qwen3.5 — M4 Pro

  Engine       Your tok/s   Community median   Delta
  ────────── ──────────── ────────────────── ────────
  lmstudio        72.6           70.1          +3.6%
  ollama          30.4           31.0          -1.9%

  Chip: Apple M4 Pro (auto-detected)
```

## Notes

- If `--chip` is not specified, asiai auto-detects your Apple Silicon chip.
- Delta shows the percentage difference between your local median and the community median.
- Positive deltas mean your setup is faster than the community average.
- Local results come from your benchmark history database (`~/.local/share/asiai/benchmarks.db` by default).



---

## /commands/config

Raw markdown: https://asiai.dev/markdown/commands/config.md
Rendered: https://asiai.dev/commands/config/

---
description: "How to configure asiai: manage engine URLs, ports, and persistent settings for your LLM benchmark setup on Mac."
---

# asiai config

Manage persistent engine configuration. Engines discovered by `asiai detect` are automatically saved to `~/.config/asiai/engines.json` for faster subsequent detection.

## Usage

```bash
asiai config show              # Show known engines
asiai config add <engine> <url> [--label NAME] [--api-key-file PATH]  # Add engine manually
asiai config remove <url>      # Remove an engine
asiai config reset             # Clear all configuration
```

## Subcommands

### show

Display all known engines with their URL, version, source (auto/manual), and last seen timestamp.

```
$ asiai config show
Known engines (3):
  ollama v0.17.7 at http://localhost:11434  (auto)  last seen 2m ago
  lmstudio v0.4.6 at http://localhost:1234  (auto)  last seen 2m ago
  omlx v0.9.2 at http://localhost:8800 [mac-mini]  (manual)  last seen 5m ago
```

### add

Manually register an engine on a non-standard port. Manual engines are never auto-pruned.

```bash
asiai config add omlx http://localhost:8800 --label desktop
asiai config add ollama http://192.0.2.10:11434 --label remote
asiai config add mtplx http://localhost:8080 --api-key-file ~/.config/asiai/keys/mtplx.key
```

### Engine API keys (`--api-key-file`)

Some servers require `Authorization: Bearer <key>` on every route (for example MTPLX started with an API key, or llama.cpp with `--api-key`). Point the engine entry at a **file containing the key** — the key itself is never stored in the config:

```json
{
  "url": "http://localhost:8080",
  "engine": "mtplx",
  "source": "manual",
  "api_key_file": "/path/to/mtplx.key"
}
```

When `api_key_file` is set, asiai sends the Bearer header on every request to **that engine's URL only** — detection probes, monitoring, the web dashboard, and benchmarks. If the file is missing, unreadable, or empty, requests are sent without the header and the server's 401 surfaces honestly. The key is never written to logs or error messages.

Keep the key file readable only by your user:

```bash
chmod 600 ~/.config/asiai/keys/mtplx.key
```

### remove

Remove an engine entry by URL.

```bash
asiai config remove http://localhost:8800
```

### reset

Delete the entire configuration file. The next `asiai detect` will re-discover engines from scratch.

## How it works

The configuration file stores engines discovered during detection:

- **Auto entries** (`source: auto`): created automatically when `asiai detect` finds a new engine. Pruned after 7 days of inactivity.
- **Manual entries** (`source: manual`): created via `asiai config add`. Never pruned automatically.

The 3-layer detection cascade in `asiai detect` uses this config as Layer 1 (fastest), followed by port scanning (Layer 2) and process detection (Layer 3). See [detect](detect.md) for details.

## Config file location

```
~/.config/asiai/engines.json
```



---

## /commands/daemon

Raw markdown: https://asiai.dev/markdown/commands/daemon.md
Rendered: https://asiai.dev/commands/daemon/

---
description: "Run asiai as a background daemon on Mac: auto-start monitoring, web dashboard, and Prometheus metrics at boot."
---

# asiai daemon

Manage background services via macOS launchd LaunchAgents.

## Services

| Service | Description | Model |
|---------|-------------|-------|
| `monitor` | Collects system + inference metrics at regular intervals | Periodic (`StartInterval`) |
| `web` | Runs the web dashboard as a persistent service | Long-running (`KeepAlive`) |

## Usage

```bash
# Monitor daemon (default)
asiai daemon start                     # Start monitoring (every 60s)
asiai daemon start --interval 30       # Custom interval
asiai daemon start --alert-webhook URL # Enable webhook alerts

# Web dashboard service
asiai daemon start web                 # Start web on 127.0.0.1:8899
asiai daemon start web --port 9000     # Custom port
asiai daemon start web --host 0.0.0.0  # Expose on network (no auth!)

# Status (shows all services)
asiai daemon status

# Stop
asiai daemon stop                      # Stop monitor
asiai daemon stop web                  # Stop web
asiai daemon stop --all                # Stop all services

# Logs
asiai daemon logs                      # Monitor logs
asiai daemon logs web                  # Web logs
asiai daemon logs web -n 100           # Last 100 lines
```

## How it works

Each service installs a separate launchd LaunchAgent plist in `~/Library/LaunchAgents/`:

- **Monitor**: runs `asiai monitor --quiet` at the configured interval (default: 60s). Data is stored in SQLite. If `--alert-webhook` is provided, alerts are POSTed on state transitions (memory pressure, thermal, engine down).
- **Web**: runs `asiai web --no-open` as a persistent process. Automatically restarts if it crashes (`KeepAlive: true`, `ThrottleInterval: 10s`).

Both services start automatically on login (`RunAtLoad: true`).

## Security

- Services run at **user level** (no root required)
- Web dashboard binds to `127.0.0.1` by default (localhost only)
- A warning is displayed when using `--host 0.0.0.0` — no authentication is configured
- Logs are stored in `~/.local/share/asiai/`



---

## /commands/detect

Raw markdown: https://asiai.dev/markdown/commands/detect.md
Rendered: https://asiai.dev/commands/detect/

---
description: Auto-detect running LLM inference engines on your Mac. 3-layer cascade — config, port scan, process detection.
---

# asiai detect

Auto-detect running inference engines using a 3-layer cascade.

## Usage

```bash
asiai detect                      # Auto-detect (3-layer cascade)
asiai detect --url http://host:port  # Scan specific URL(s) only
```

## Output

```
Detected engines:

  ● ollama 0.17.4
    URL: http://localhost:11434

  ● lmstudio 0.4.5
    URL: http://localhost:1234
    Running: 1 model(s)
      - qwen3.5-35b-a3b  MLX

  ● omlx 0.9.2
    URL: http://localhost:8800
```

## How it works: 3-layer detection

asiai uses a cascade of three detection layers, from fastest to most thorough:

### Layer 1: Config (fastest, ~100ms)

Reads `~/.config/asiai/engines.json` — engines discovered in previous runs. This catches engines on non-standard ports (e.g., oMLX on 8800) without rescanning.

### Layer 2: Port scan (~200ms)

Scans default ports plus an extended range:

| Port | Engine |
|------|--------|
| 11434 | Ollama |
| 1234 | LM Studio |
| 8080 | mlx-lm or llama.cpp |
| 8000-8009 | oMLX or vllm-mlx |
| 52415 | Exo |

### Layer 3: Process detection (fallback)

Uses `ps` and `lsof` to find engine processes listening on any port. Catches engines running on completely unexpected ports.

### Auto-persist

Any engine discovered in Layer 2 or 3 is automatically saved to the config file (Layer 1) for faster detection next time. Auto-discovered entries are pruned after 7 days of inactivity.

When multiple engines share a port (e.g., mlx-lm and llama.cpp on 8080), asiai uses API endpoint probing to identify the correct engine.

## Explicit URLs

When using `--url`, only the specified URLs are scanned. No config is read or written — useful for one-off checks.

```bash
asiai detect --url http://192.0.2.10:11434,http://localhost:8800
```

## See also

- [config](config.md) — Manage persistent engine configuration



---

## /commands/doctor

Raw markdown: https://asiai.dev/markdown/commands/doctor.md
Rendered: https://asiai.dev/commands/doctor/

---
description: "Diagnose LLM inference issues on Mac: asiai doctor checks engine health, port conflicts, model loading, and GPU status."
---

# asiai doctor

Diagnose installation, engines, system health, and database.

## Usage

```bash
asiai doctor
```

## Output

```
Doctor

  System
    ✓ Apple Silicon       Mac Mini M4 Pro — Apple M4 Pro
    ✓ RAM                 64 GB total, 42% used
    ✓ Memory pressure     normal
    ✓ Thermal             nominal (100%)

  Engine
    ✓ Ollama              v0.17.5 — 1 model(s): qwen3.5:35b-a3b
    ✓ Ollama config       host=0.0.0.0:11434, num_parallel=1 (default), ...
    ✓ LM Studio           v0.4.6 — 1 model(s): qwen3.5-35b-a3b
    ✗ mlx-lm              not installed
    ✗ llama.cpp           not installed
    ✗ vllm-mlx            not installed

  Database
    ✓ SQLite              2.4 MB, last entry: 1m ago

  Daemon
    ✓ Monitoring daemon   running PID 1234
    ✓ Web dashboard       not installed

  Alerting
    ✓ Webhook URL         https://hooks.slack.com/services/...
    ✓ Webhook reachable   HTTP 200

  9 ok, 0 warning(s), 3 failed
```

## Checks

- **System**: Apple Silicon detection, RAM, memory pressure, thermal state
- **Engine**: Reachability and version for all 7 supported engines; Ollama runtime parameters (host, num_parallel, max_loaded_models, keep_alive, flash_attention)
- **Database**: SQLite schema version, size, last entry timestamp
- **Daemon**: LaunchAgent status for monitor and web services
- **Alerting**: Webhook URL configuration and connectivity



---

## /commands/leaderboard

Raw markdown: https://asiai.dev/markdown/commands/leaderboard.md
Rendered: https://asiai.dev/commands/leaderboard/

---
description: "Browse and query the asiai community leaderboard: compare benchmark results across Apple Silicon chips and inference engines."
---

# asiai leaderboard

Browse community benchmark data from the asiai network.

## Usage

```bash
asiai leaderboard [options]
```

## Options

| Option | Description |
|--------|-------------|
| `--chip CHIP` | Filter by Apple Silicon chip (e.g., `M4 Pro`, `M2 Ultra`) |
| `--model MODEL` | Filter by model name |
| `--agentic DIR` | Render the local **decision-tier** view from a directory of `--agentic-mode` result JSON (no network) instead of the community feed |
| `--grid` | With `--agentic`: show the full **archive grid** (every column) instead of the tier view |

## Example

```bash
asiai leaderboard --chip "M4 Pro"
```

### Local agentic results — decision tiers

Render your own `asiai bench --agentic-mode --agentic-output` results, grouped by
deterministic gates into tiers (★ best validated throughput · ✓ viable · ⚠ reserve
· ✗ eliminated), per `(machine, power mode)` block:

```bash
asiai leaderboard --agentic ./my-bench-results/
asiai leaderboard --agentic ./my-bench-results/ --grid   # full archive table
```

```
Agentic bench — decision tiers
  ★ best validated throughput · ✓ viable · ⚠ reserve · ✗ eliminated.
  gates: valid≥80% · ttft≤1500ms (hard≤3000) · reuse>0.

  ▰ M5 · Q4_K_S · Apple M5 Max · powermode 2
     model · engine                 dec    peak    50K    ttft  reuse   t/s/W   RAMg  val%
  ★ TIER 1 — winner + fast
  ★  Qwopus-35B · llamacpp b9430 ▲MTP  123.3  127.5   83.8     67    0.8   1.590   29.0   100
```

Each row is self-describing from schema `agentic-v4`: the machine, chip, power
mode and engine version are read from the JSON, so the table needs no filename
parsing or hardcoded version map. Gates match the community ranking
(`valid ≥ 80%`); `★` ranks throughput only — the final pick also weighs output
quality. A throttled (power mode 0) run is never tiered against a High Power one.

```
  Community Leaderboard — M4 Pro

  Model                  Engine      tok/s (median)   Runs   Contributors
  ──────────────────── ────────── ──────────────── ────── ──────────────
  qwen3.5:35b-a3b       ollama          30.4          42         12
  qwen3.5:35b-a3b       lmstudio        72.6          38         11
  llama3.3:70b           exo            18.2          15          4
  gemma2:9b              mlx-lm        105.3          27          9

  Source: api.asiai.dev — 122 results
```

## Notes

- Requires the community API at `api.asiai.dev`.
- Results are anonymized. No personal or machine-identifying data is shared.
- Contribute your own results with `asiai bench --share`.



---

## /commands/mcp

Raw markdown: https://asiai.dev/markdown/commands/mcp.md
Rendered: https://asiai.dev/commands/mcp/

---
description: MCP server exposing 11 tools for AI agents to monitor inference engines, run benchmarks and get hardware-aware recommendations.
---

# asiai mcp

Start the MCP (Model Context Protocol) server, enabling AI agents to monitor and benchmark your inference infrastructure.

## Usage

```bash
asiai mcp                          # stdio transport (Claude Code)
asiai mcp --transport sse          # SSE transport (network agents)
asiai mcp --transport sse --port 9000
```

## Options

| Option | Description |
|--------|-------------|
| `--transport` | Transport protocol: `stdio` (default), `sse`, `streamable-http` |
| `--host` | Bind address (default: `127.0.0.1`) |
| `--port` | Port for SSE/HTTP transport (default: `8900`) |
| `--register` | Opt-in registration with asiai agent network (anonymous) |

## Tools (11)

| Tool | Description | Read-only |
|------|-------------|-----------|
| `check_inference_health` | Quick health check: engines up/down, memory pressure, thermal, GPU | Yes |
| `get_inference_snapshot` | Full system snapshot with all metrics | Yes |
| `list_models` | List all loaded models across engines | Yes |
| `detect_engines` | Re-scan for inference engines | Yes |
| `run_benchmark` | Run a benchmark or cross-model comparison (rate-limited to 1/min) | No |
| `get_recommendations` | Hardware-aware engine/model recommendations | Yes |
| `diagnose` | Run diagnostic checks (like `asiai doctor`) | Yes |
| `get_metrics_history` | Query historical metrics (1-168 hours) | Yes |
| `get_benchmark_history` | Query past benchmark results with filters | Yes |
| `compare_engines` | Compare engine performance for a model with verdict; supports multi-model comparison from history | Yes |
| `refresh_engines` | Re-detect engines without restarting the server | Yes |

## Resources (3)

| Resource | URI | Description |
|----------|-----|-------------|
| System Status | `asiai://status` | Current system health (memory, thermal, GPU) |
| Models | `asiai://models` | All loaded models across engines |
| System Info | `asiai://system` | Hardware info (chip, RAM, cores, OS, uptime) |

## Claude Code integration

Add to your Claude Code MCP config (`~/.claude/claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "asiai": {
      "command": "asiai",
      "args": ["mcp"]
    }
  }
}
```

Then ask Claude: *"Check my inference health"* or *"Compare Ollama vs LM Studio for qwen3.5"*.

## Benchmark cards

The `run_benchmark` tool supports card generation via the `card` parameter. When `card=true`, a 1200x630 SVG benchmark card is generated and `card_path` is returned in the response.

```json
{"tool": "run_benchmark", "arguments": {"model": "qwen3.5", "card": true}}
```

Cross-model comparison (mutually exclusive with `model`, max 8 slots):

```json
{"tool": "run_benchmark", "arguments": {"compare": ["qwen3.5:4b", "deepseek-r1:7b"], "card": true}}
```

CLI equivalent for PNG + sharing:

```bash
asiai bench --quick --card --share    # Quick bench + card + share (~15s)
```

See the [Benchmark Card](../benchmark-card.md) page for details.

## Agent registration

Join the asiai agent network to get community features (leaderboard, comparison, percentile stats):

```bash
asiai mcp --register                  # Register on first run, heartbeat on subsequent runs
asiai unregister                      # Remove local credentials
```

Registration is **opt-in and anonymous** — only hardware info (chip, RAM) and engine names are sent. No IP, hostname, or personal data is stored. Credentials are saved in `~/.local/share/asiai/agent.json` (chmod 600).

On subsequent `asiai mcp --register` calls, a heartbeat is sent instead of re-registering. If the API is unreachable, the MCP server starts normally without registration.

Check your registration status with `asiai version`.

## Network agents

For agents on other machines (e.g., monitoring a headless Mac Mini):

```bash
asiai mcp --transport sse --host 0.0.0.0 --port 8900
```

See the [Agent Integration guide](../agent.md) for detailed setup instructions.



---

## /commands/models

Raw markdown: https://asiai.dev/markdown/commands/models.md
Rendered: https://asiai.dev/commands/models/

---
description: "List all loaded LLM models across engines: see VRAM usage, quantization, context length, and format for each model."
---

# asiai models

List loaded models across all detected engines.

## Usage

```bash
asiai models
```

## Output

```
ollama  v0.17.5  http://localhost:11434
  ● qwen3.5:35b-a3b                             26.0 GB Q4_K_M

lmstudio  v0.4.6  http://localhost:1234
  ● qwen3.5-35b-a3b                              9.2 GB    MLX
```

Shows engine version, model name, VRAM usage (when available), format, and quantization level for each engine.

VRAM is reported natively by Ollama and LM Studio. For other engines, asiai estimates memory usage via `ri_phys_footprint` (the macOS physical footprint, same as Activity Monitor). Estimated values are labeled "(est.)".



---

## /commands/monitor

Raw markdown: https://asiai.dev/markdown/commands/monitor.md
Rendered: https://asiai.dev/commands/monitor/

---
description: Continuous GPU utilization, thermal state and memory pressure monitoring for Apple Silicon. No sudo required.
---

# asiai monitor

System and inference metrics snapshot, stored in SQLite.

## Usage

```bash
asiai monitor [options]
```

## Options

| Option | Description |
|--------|-------------|
| `-w, --watch SEC` | Refresh every SEC seconds |
| `-q, --quiet` | Collect and store without output (for daemon use) |
| `-H, --history PERIOD` | Show history (e.g., `24h`, `1h`) |
| `-a, --analyze HOURS` | Comprehensive analysis with trends |
| `-c, --compare TS TS` | Compare two timestamps |
| `--alert-webhook URL` | POST alerts to webhook URL on state transitions |

## Output

```
System
  Uptime:    3d 12h
  CPU Load:  2.45 / 3.12 / 2.89  (1m / 5m / 15m)
  Memory:    45.2 GB / 64.0 GB  71%
  Pressure:  normal
  Thermal:   nominal  (100%)

GPU
  Utilization: 45%  (renderer 44%, tiler 45%)
  Memory:      24.2 GB in use / 48.0 GB allocated

Power
  GPU: 12.6W  CPU: 4.4W  ANE: 0.0W  DRAM: 5.2W
  Total: 22.2W  (IOReport, no sudo)

Inference  ollama 0.17.4
  Models loaded: 1  VRAM total: 26.0 GB

  Model                                        VRAM   Format  Quant
  ──────────────────────────────────────── ────────── ──────── ──────
  qwen3.5:35b-a3b                            26.0 GB     gguf Q4_K_M
```

Power monitoring uses Apple's IOReport Energy Model to read GPU, CPU, ANE and DRAM power consumption — no sudo required. See [Methodology](../methodology.md#power-measurement) for validation details.

## Alert webhooks

When `--alert-webhook URL` is provided, asiai will POST a JSON alert to the webhook URL whenever a **state transition** is detected:

| Alert type | Trigger | Severity |
|------------|---------|----------|
| `mem_pressure_warn` | Memory pressure: normal → warn | warning |
| `mem_pressure_critical` | Memory pressure: normal/warn → critical | critical |
| `thermal_degraded` | Thermal level: nominal → fair/serious/critical | warning/critical |
| `engine_down` | Engine was reachable, now unreachable | critical |

Alerts use a **5-minute cooldown** per type to prevent spam. Each alert is stored in SQLite for history.

### Webhook payload

```json
{
    "alert": "mem_pressure_warn",
    "severity": "warning",
    "ts": 1741350000,
    "host": "macmini.local",
    "message": "Memory pressure changed: normal → warn",
    "details": {
        "mem_pressure": "warn",
        "mem_used": 54000000000,
        "mem_total": 68719476736
    },
    "source": "asiai/0.7.0"
}
```

### Usage with daemon

```bash
asiai daemon start monitor --alert-webhook https://hooks.slack.com/services/...
```

## Data storage

All snapshots are stored in SQLite (`~/.local/share/asiai/metrics.db`) with 90-day automatic retention.



---

## /commands/recommend

Raw markdown: https://asiai.dev/markdown/commands/recommend.md
Rendered: https://asiai.dev/commands/recommend/

---
description: Hardware-aware model recommendations based on your Mac's RAM, GPU cores and thermal headroom.
---

# asiai recommend

Get engine recommendations for your hardware and use case.

## Usage

```bash
asiai recommend [options]
```

## Options

| Option | Description |
|--------|-------------|
| `--model MODEL` | Model to get recommendations for |
| `--use-case USE_CASE` | Optimize for: `throughput`, `latency`, or `efficiency` |
| `--community` | Include community benchmark data in recommendations |
| `--db PATH` | Path to local benchmark database |

## Data sources

Recommendations are built from the best available data, in priority order:

1. **Local benchmarks** — your own runs on your hardware
2. **Community data** — aggregated results from similar chips (with `--community`)
3. **Heuristics** — built-in rules when no benchmark data is available

## Confidence levels

| Level | Criteria |
|-------|----------|
| High | 5 or more local benchmark runs |
| Medium | 1 to 4 local runs, or community data available |
| Low | Heuristic-based, no benchmark data |

## Example

```bash
asiai recommend --model qwen3.5 --use-case throughput
```

```
  Recommendation: qwen3.5 — M4 Pro — throughput

  #   Engine       tok/s    Confidence   Source
  ── ────────── ──────── ──────────── ──────────
  1   lmstudio    72.6     high         local (5 runs)
  2   ollama      30.4     high         local (5 runs)
  3   exo         18.2     medium       community

  Tip: lmstudio is 2.4x faster than ollama for this model.
```

## Notes

- Run `asiai bench` first for the most accurate recommendations.
- Use `--community` to fill gaps when you haven't benchmarked a specific engine locally.
- The `efficiency` use case factors in power consumption (requires `--power` data from previous benchmarks).



---

## /commands/setup

Raw markdown: https://asiai.dev/markdown/commands/setup.md
Rendered: https://asiai.dev/commands/setup/

---
description: "Quick setup for asiai: configure engines, test connections, and verify your Apple Silicon Mac is ready for LLM benchmarking."
---

# asiai setup

Interactive setup wizard for first-time users. Detects your hardware, checks for inference engines, and suggests next steps.

## Usage

```bash
asiai setup
```

## What it does

1. **Hardware detection** — identifies your Apple Silicon chip and RAM
2. **Engine scan** — checks for installed inference engines (Ollama, LM Studio, mlx-lm, llama.cpp, oMLX, vllm-mlx, Exo)
3. **Model check** — lists loaded models across all detected engines
4. **Daemon status** — shows whether the monitoring daemon is running
5. **Next steps** — suggests commands based on your setup state

## Example output

```
  Setup Wizard

  Hardware:  Apple M4 Pro, 64 GB RAM

  Engines:
    ✓ ollama (v0.17.7) — 3 models loaded
    ✓ lmstudio (v0.4.5) — 1 model loaded

  Daemon: running (monitor + web)

  Suggested next steps:
    • asiai bench              Run your first benchmark
    • asiai monitor --watch    Watch metrics live
    • asiai web                Open the dashboard
```

## When no engines are found

If no engines are detected, setup provides installation guidance:

```
  Engines:
    No inference engines detected.

  To get started, install an engine:
    brew install ollama && ollama serve
    # or download LM Studio from https://lmstudio.ai
```



---

## /commands/tui

Raw markdown: https://asiai.dev/markdown/commands/tui.md
Rendered: https://asiai.dev/commands/tui/

---
description: "Terminal UI for asiai: monitor LLM inference engines in real-time with an interactive dashboard in your terminal."
---

# asiai tui

Interactive terminal dashboard with auto-refresh.

## Usage

```bash
asiai tui
```

## Requirements

Requires the `tui` extra:

```bash
pip install asiai[tui]
```

This installs [Textual](https://textual.textualize.io/) for the terminal UI.

## Features

- Real-time system metrics (CPU, memory, thermal)
- Engine status and loaded models
- Auto-refresh with configurable interval



---

## /commands/version

Raw markdown: https://asiai.dev/markdown/commands/version.md
Rendered: https://asiai.dev/commands/version/

---
description: "Check asiai version, Python environment, and agent registration status with a single command."
---

# asiai version

Display version and system information.

## Usage

```bash
asiai version
asiai --version
```

## Output

The `version` subcommand shows enriched system context:

```
asiai 1.0.1
  Apple M4 Pro, 64 GB RAM
  Engines: ollama, lmstudio
  Daemon: monitor, web
```

The `--version` flag shows only the version string:

```
asiai 1.0.1
```

## Use cases

- Quick system check in issues and bug reports
- Agent context gathering (chip, RAM, available engines)
- Scripting: `VERSION=$(asiai version | head -1)`



---

## /commands/web

Raw markdown: https://asiai.dev/markdown/commands/web.md
Rendered: https://asiai.dev/commands/web/

---
description: Real-time LLM monitoring dashboard in your browser. GPU metrics, engine health, performance history. No setup required.
---

# asiai web

Launch the web dashboard for visual monitoring and benchmarking.

## Usage

```bash
asiai web
asiai web --port 9000
asiai web --host 0.0.0.0
asiai web --no-open
```

## Options

| Option | Default | Description |
|--------|---------|-------------|
| `--port` | `8899` | HTTP port to listen on |
| `--host` | `127.0.0.1` | Host to bind to |
| `--no-open` | | Don't open the browser automatically |
| `--db` | `~/.local/share/asiai/asiai.db` | Path to the SQLite database |

## Requirements

The web dashboard requires additional dependencies:

```bash
pip install asiai[web]
# or install everything:
pip install asiai[all]
```

## Pages

### Dashboard (`/`)

System overview with engine status, loaded models, memory usage, and last benchmark results.

### Benchmark (`/bench`)

Run cross-engine benchmarks directly from the browser:

- **Quick Bench** button — 1 prompt, 1 run, ~15 seconds
- Advanced options: engines, prompts, runs, context-size (4K/16K/32K/64K), power
- Live progress via SSE
- Results table with winner highlighting
- Throughput and TTFT charts
- **Shareable card** — auto-generated after benchmark (PNG via API, SVG fallback)
- **Share section** — copy link, download PNG/SVG, share on X/Reddit, export JSON

### History (`/history`)

Visualize benchmark and system metrics over time:

- System charts: CPU load, Memory %, GPU utilization (with renderer/tiler breakdown)
- Engine activity: TCP connections, requests processing, KV cache usage %
- Benchmark charts: throughput (tok/s) and TTFT per engine
- Process metrics: engine CPU % and RSS memory during benchmark runs
- Filter by time range (1h / 24h / 7d / 30d / 90d) or custom date range
- Data table with context-size indication (e.g., "code (64K ctx)")

### Monitor (`/monitor`)

Real-time system monitoring with 5-second refresh:

- CPU load sparkline
- Memory gauge
- Thermal state
- Loaded models list

### Doctor (`/doctor`)

Interactive health check for system, engines, and database. Same checks as `asiai doctor` with a visual interface.

## API Endpoints

The web dashboard exposes REST API endpoints for programmatic access.

### `GET /api/status`

Lightweight health check. Cached 10s, responds in < 500ms.

```json
{
  "status": "ok",
  "ts": 1709700000,
  "uptime": 86400,
  "engines": {"ollama": true, "lmstudio": false},
  "memory_pressure": "normal",
  "thermal_level": "nominal"
}
```

Status values: `ok` (all engines reachable), `degraded` (some down), `error` (all down).

### `GET /api/snapshot`

Full system + engine snapshot. Cached 5s. Includes CPU load, memory, thermal state, and per-engine status with loaded models.

### `GET /api/benchmarks`

Benchmark results with filters. Returns per-run data including tok/s, TTFT, power, context_size, engine_version.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `hours` | `168` | Time range in hours (0 = all) |
| `model` | | Filter by model name |
| `engine` | | Filter by engine name |
| `since` / `until` | | Unix timestamp range (overrides hours) |

### `GET /api/engine-history`

Engine status history (reachability, TCP connections, KV cache, tokens predicted).

| Parameter | Default | Description |
|-----------|---------|-------------|
| `hours` | `168` | Time range in hours |
| `engine` | | Filter by engine name |

### `GET /api/benchmark-process`

Process-level CPU and memory metrics from benchmark runs (7-day retention).

| Parameter | Default | Description |
|-----------|---------|-------------|
| `hours` | `168` | Time range in hours |
| `engine` | | Filter by engine name |

### `GET /api/metrics`

Prometheus exposition format. Gauges covering system, engine, model, and benchmark metrics.

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'asiai'
    static_configs:
      - targets: ['localhost:8899']
    metrics_path: '/api/metrics'
    scrape_interval: 30s
```

Metrics include:

| Metric | Type | Description |
|--------|------|-------------|
| `asiai_cpu_load_1m` | gauge | CPU load average (1 min) |
| `asiai_memory_used_bytes` | gauge | Memory used |
| `asiai_thermal_speed_limit_pct` | gauge | CPU speed limit % |
| `asiai_engine_reachable{engine}` | gauge | Engine reachability (0/1) |
| `asiai_engine_models_loaded{engine}` | gauge | Models loaded count |
| `asiai_engine_tcp_connections{engine}` | gauge | Established TCP connections |
| `asiai_engine_requests_processing{engine}` | gauge | Requests currently processing |
| `asiai_engine_kv_cache_usage_ratio{engine}` | gauge | KV cache fill ratio (0-1) |
| `asiai_engine_tokens_predicted_total{engine}` | counter | Cumulative tokens predicted |
| `asiai_model_vram_bytes{engine,model}` | gauge | VRAM per model |
| `asiai_bench_tok_per_sec{engine,model}` | gauge | Last benchmark tok/s |

## Notes

- The dashboard binds to `127.0.0.1` by default (localhost only)
- Use `--host 0.0.0.0` to expose on the network (e.g., for remote monitoring)
- Port `8899` is chosen to avoid conflicts with inference engine ports



---

## /dev-quality-benchmarks

Raw markdown: https://asiai.dev/markdown/dev-quality-benchmarks.md
Rendered: https://asiai.dev/dev-quality-benchmarks/

---
description: Dev-quality and multilingual-retention benchmark results on Apple Silicon — tool-call reliability (the JSON arg-truncation / empty-object bug), agentic error-recovery, thinking discipline, and language retention. Deterministic, no LLM judge needed for the core signal. A living results page.
---

# Dev-Quality & Language Benchmarks

Throughput is not quality. A model can decode fast and still be unusable for
agentic coding — it truncates tool-call arguments, loops on errors, or its
finetune quietly broke another language. This page reports real
`asiai bench --code` and `asiai bench --language` results: **deterministic**
signals (no LLM judge needed for the core) that measure whether a model actually
works, not how fast it emits tokens.

> **Living document.** Numbers are refreshed as model revisions, engines and
> templates change. Each block names the exact model file and serving config so a
> result is reproducible.

## What is measured

`asiai bench --code` (deterministic, no judge):

- **tool-call** — an 8-turn agentic file-editing session under accumulating
  context. Scores tool-call emission, JSON validity, non-truncation, correct
  tool, schema conformance, and the **empty-object bug**: the `|items` template
  truncation that collapses an `edit_file.edits` array to `{}` / `[]`.
- **tool-call-stress** — the same, harder: deeper context, 8–10-element edit
  arrays, JSON-escaping pressure (newlines, quotes, backslashes, unicode). Used
  to tell apart models that ace the baseline.
- **recovery** — inject a synthetic tool error mid-session; score a corrective
  action vs. a stuck loop (re-emitting the failing call).
- **thinking** — thinking-mode discipline: no `<think>` leak into content,
  non-empty output at a short budget, and `enable_thinking=false` honoured.
- **coding** / **coding-hard** *(optional judge)* — multi-turn coding tasks
  graded 1–5 by an LLM judge at `--judge-url` (any OpenAI-compatible endpoint).

`asiai bench --instruct` (deterministic instruction-following):

- **verifiable** — IFEval-style single-turn prompts with programmatically-checkable
  instructions (word/sentence/section counts, keywords, JSON-only, casing, no
  commas, end phrase, title in `<<>>`, language…). Reported as strict/loose
  accuracy at prompt-level and instruction-level — the public-leaderboard format.
  asiai-native reimplementation of the IFEval paradigm (Zhou et al. 2023); no
  IFEval code or data is vendored.
- **research-brief** — an agentic task: research several topics via tools, then
  write a multi-section briefing, then a secondary tool action (save) **last**.
  Does the model produce the primary briefing, or do the tool work and return only
  the secondary-step confirmation? A model can ace tool-call reliability and still
  skip the main deliverable — scored deterministically by checking the required
  sections appear after the tool turns. **order-control** swaps the order
  (secondary first) as the diagnostic.
- **loop-search** — an ambiguous-search trap: a deep warmup over clear topics,
  then a target fact that `web_search` can never confirm (semantic reformulations
  of the query collapse to one answer). Scores whether the model accepts the
  ambiguity and delivers (sober) or re-issues equivalent queries until a
  no-progress cap halts it (perfectionist), plus an output-token-collapse signal.
  Two modes (`short` sub-1KB result / `unconfirmable` plausible-but-missing-fact).
  This is the failure mode that single-turn IFEval and research-brief don't
  surface.

`asiai bench --language <code>` (deterministic, 8 languages):

- **adherence** — does the model stay in the target language? (target vs. English
  function-word ratio for Latin scripts; target-script character ratio for
  ja/ko/zh).
- **diacritics** — trap prompts whose correct answer must contain specific
  accented tokens (`café`, `préféré`); an ASCII-stripped answer fails.

`asiai bench --thinking-ablation` (cost/benefit of the thinking config):

Runs one representative multi-turn agentic file-editing load (the tool-call-stress
turns, which accumulate context) under three thinking configurations and reports
the trade-off that decides the production setting:

- **enable-off** — no reasoning generated (preserve is moot).
- **enable-on-preserve-on** — reasoning kept in multi-turn history: coherent
  across turns, but context grows.
- **enable-on-preserve-off** — reasoning generated each turn but stripped from
  history: cheaper context, fresher each turn, less loop amplification.

Per config it reports tool-call quality, latency per turn, and prompt (context)
tokens at turn N. The history is rebuilt *with* `reasoning_content` so
`preserve_thinking` has a real effect (without that, preserve on vs off would be
a no-op). Schema `thinking-ablation-v1`; takes a single `--url` target.

All four modes are JSON-only and compare across models by diffing the output.

## Worked example — Qwen3.6-35B-A3B vs Qwopus3.6-35B-A3B vs Qwen3.6-27B dense

A finetune (`Qwopus3.6`, an Opus-distilled finetune of the `Qwen3.6-35B-A3B` MoE)
vs. its base, vs. a dense model half its size. Same llama.cpp, **same chat
template held constant** (only the model file swapped), thinking disabled, 3
repeats. Apple Silicon M5 Max, High Power Mode. Measured on dataset `code-v1` (9-turn stress suite). The suite gained a large-payload cell in `code-v2` (11 turns): the empty-object figures below are RAW COUNTS, so compare them only against other `code-v1` runs.

### Tool-call reliability

| model · quant | tool-call clean | empty-object bug | under stress |
|---|--:|--:|--:|
| Qwen3.6-35B-A3B base · Q4 / Q5 | 87.5% | **3** | 87.5% · **3 bugs** |
| **Qwopus3.6-35B-A3B · Q4** | **100%** | **0** | **100% · 0** |
| Qwen3.6-27B dense · Q5 | 100% | 0 | 88.9% · **3 bugs** |

- **The base 35B MoE has a residual tool-call defect the template fix does not
  fully close.** It collapses `edit_file.edits` to the empty-object bug 3/3 on a
  deep-context turn — at **both Q4 and Q5** quants (so it is a generation
  behaviour, not quantisation). The community `froggeric` template, which fixes
  the `|items` bug on simple calls, does not save the base MoE deep in context.
- **The Opus-distilled finetune repairs it completely** — 0 bugs, 100% clean —
  and at a *lower* quant (Q4 vs Q5), which makes the win stronger.
- **Under stress, the finetune is the more robust agent than the dense 27B**:
  the 27B cracks (3 empty-object bugs on the harder suite) while the finetune
  stays at 0. They tie on the baseline; the stress suite separates them.

### Code correctness (LLM-judged hard tasks)

On two trickier multi-turn coding tasks they **split**: on a sliding-window rate
limiter both handle the boundary/eviction edge cases; on an expression evaluator
the **dense 27B gets operator precedence right** (`-2**2 == -4`, unary minus as a
proper operator) while the **finetune does not** (it folds the unary minus into
the number → `4.0`). Tool-call robustness and algorithmic correctness are
*different* axes — measure both.

### Language retention

Running `--language fr` on the finetune and its base, same quant:

| model | adherence | diacritic traps | ASCII-stripped |
|---|--:|--:|--:|
| Qwen3.6-35B-A3B base | 100% | 4/4 | 0 |
| **Qwopus3.6-35B-A3B** | 100% | 4/4 | 0 |

**Zero French regression.** The coding-oriented finetune kept the base model's
French intact (adherence, diacritics, no ASCII-stripping) — a task-specific
finetune did *not* cost another language, which is worth verifying rather than
assuming.

### Perfectionist research loop (loop-search)

`research-brief` saturates at 100% for every model here, so it does not
discriminate the *perfectionist loop* that breaks real agents. The `loop-search`
scenario does. Across a sweep of dense 27B and MoE 35B-A3B configs (M5, llama.cpp
b9430, thinking on/off, both ambiguity modes):

- The **35B-A3B MoE loops** — it re-issues semantically-equivalent searches on an
  unconfirmable fact until a no-progress guardrail halts it, instead of accepting
  the uncertainty and delivering. It does so in **both Q4 and Q8** (architectural,
  not a quant artefact), for the base and the Opus-distilled finetune alike.
- The **dense 27B never loops** (Q4 / Q5 / Q8): it accepts the ambiguous result
  and writes the briefing.

For an agentic harness such as NousResearch's Hermes Agent this is the deciding
signal: the loop-resistant dense model is the safer main even when a faster MoE
exists — throughput buys nothing if the agent spirals on one ambiguous step. It is
also the inverse lesson of the tool-call result above (where the MoE finetune was
the *more* robust agent): **fitness is per-failure-mode, so measure several.**

## How to read this

- **Verdict-first, not speed-first.** These are correctness/reliability signals.
  For throughput, see the [Agentic Benchmarks](agentic-benchmarks.md).
- **Deterministic core, optional judge.** tool-call / recovery / thinking /
  adherence / diacritics need no LLM judge — they are reproducible. The
  `coding`/`fluency` grades are LLM-judged (subjective, optional).
- **Compare within a controlled change.** The example holds the template constant
  and varies only the model, so a difference is the model's, not the harness's.

## Methodology & caveats

- `asiai bench --code` / `--language`, thinking disabled
  (`chat_template_kwargs.enable_thinking=false`), one engine resident at a time.
- **Quant differs across the example** (the finetune Q4 vs the Qwen models Q5):
  the headline empty-object bug is template/generation-driven and was confirmed
  at **both** quants for the base, so quant does not explain the gap — and the
  finetune wins from the lower quant.
- **The code-quality judge is not strictly blind** here (a frontier model read
  the transcripts on the merits); the deterministic tool-call/stress numbers are
  objective.
- **Recovery is weight-sensitive**, not a clean cross-model signal — the headline
  is the tool-call/empty-object reliability, which is stable across repeats.

See also: [Agentic Benchmarks](agentic-benchmarks.md) ·
[Benchmark methodology](methodology.md) · [Metrics spec](metrics-spec.md).



---

## /engines/exo

Raw markdown: https://asiai.dev/markdown/engines/exo.md
Rendered: https://asiai.dev/engines/exo/

---
description: "Exo distributed LLM inference: benchmark multiple Macs together, port 52415, cluster setup and performance."
---

# Exo

Exo enables distributed LLM inference by pooling VRAM across multiple Apple Silicon Macs on your local network, serving on port 52415. It lets you run 70B+ parameter models that would not fit on a single machine, with automatic peer discovery and an OpenAI-compatible API.

[Exo](https://github.com/exo-explore/exo) enables distributed inference across multiple Apple Silicon devices. Run large models (70B+) by pooling VRAM from several Macs.

## Setup

```bash
pip install exo-inference
exo
```

Or install from source:

```bash
git clone https://github.com/exo-explore/exo.git
cd exo && pip install -e .
exo
```

## Details

| Property | Value |
|----------|-------|
| Default port | 52415 |
| API type | OpenAI-compatible |
| VRAM reporting | Yes (aggregated across cluster nodes) |
| Model format | GGUF / MLX |
| Detection | Auto via DEFAULT_URLS |

## Benchmarking

```bash
asiai bench --engines exo -m llama3.3:70b
```

Exo is benchmarked like any other engine. asiai auto-detects it on port 52415.

## Notes

- Exo discovers peer nodes automatically on the local network.
- VRAM displayed in asiai reflects the total memory aggregated across all cluster nodes.
- Large models that don't fit on a single Mac can run seamlessly across the cluster.
- Start `exo` on each Mac in the cluster before running benchmarks.

## See also

Compare engines with `asiai bench --engines exo` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/llamacpp

Raw markdown: https://asiai.dev/markdown/engines/llamacpp.md
Rendered: https://asiai.dev/engines/llamacpp/

---
description: "llama.cpp server on Mac: low-level control, port 8080, KV cache metrics, and benchmark results on Apple Silicon."
---

# llama.cpp

llama.cpp is the foundational C++ inference engine for GGUF models, offering maximum low-level control over KV cache, thread count, and context size on port 8080. It powers Ollama's backend but can be run standalone for fine-grained tuning on Apple Silicon.

[llama.cpp](https://github.com/ggml-org/llama.cpp) is a high-performance C++ inference engine supporting GGUF models.

## Setup

```bash
brew install llama.cpp
llama-server -m model.gguf
```

## Details

| Property | Value |
|----------|-------|
| Default port | 8080 |
| API type | OpenAI-compatible |
| VRAM reporting | No |
| Model format | GGUF |
| Detection | `/health` + `/props` endpoints or `lsof` process detection |

## Notes

- llama.cpp shares port 8080 with mlx-lm. asiai detects it via the `/health` and `/props` endpoints.
- The server can be started with custom context sizes and thread counts for tuning.

## See also

Compare engines with `asiai bench --engines llamacpp` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/lmstudio

Raw markdown: https://asiai.dev/markdown/engines/lmstudio.md
Rendered: https://asiai.dev/engines/lmstudio/

---
description: "LM Studio benchmark on Apple Silicon: fastest MLX engine, port 1234 setup, VRAM usage, and how it compares to Ollama."
---

# LM Studio

LM Studio is the fastest MLX inference engine on Apple Silicon, serving models on port 1234 with an OpenAI-compatible API. On M4 Pro 64GB, it reaches 130 tok/s on Qwen3-Coder-30B (MLX), nearly 2x faster than Ollama's llama.cpp backend for MoE models.

[LM Studio](https://lmstudio.ai) provides an OpenAI-compatible API with a GUI for model management.

## Setup

```bash
brew install --cask lm-studio
```

Start the local server from the LM Studio app, then load a model.

## Details

| Property | Value |
|----------|-------|
| Default port | 1234 |
| API type | OpenAI-compatible |
| VRAM reporting | Yes (via `lms ps --json` CLI) |
| Model format | GGUF, MLX |
| Detection | `/lms/version` endpoint or app bundle plist |

## VRAM reporting

Since v0.7.0, asiai retrieves VRAM usage from the LM Studio CLI (`~/.lmstudio/bin/lms ps --json`). This provides accurate model size data that the OpenAI-compatible API does not expose.

If the `lms` CLI is not installed or unavailable, asiai gracefully falls back to reporting VRAM as 0 (same behavior as before v0.7.0).

## Notes

- LM Studio supports both GGUF and MLX model formats.
- Version detection uses the `/lms/version` API endpoint, with a fallback to the app bundle plist on disk.
- Model names typically use the HuggingFace format (e.g., `gemma-2-9b-it`).

## See also

See how LM Studio compares: [Ollama vs LM Studio benchmark](../ollama-vs-lmstudio.md)



---

## /engines/mlxlm

Raw markdown: https://asiai.dev/markdown/engines/mlxlm.md
Rendered: https://asiai.dev/engines/mlxlm/

---
description: "mlx-lm server benchmark on Mac: optimal for MoE models, port 8080 config, and Apple Silicon performance data."
---

# mlx-lm

mlx-lm is Apple's reference MLX inference server, running models natively on Metal GPU via port 8080. It is particularly efficient for MoE (Mixture of Experts) models on Apple Silicon, leveraging unified memory for zero-copy model loading.

[mlx-lm](https://github.com/ml-explore/mlx-examples) runs models natively on Apple MLX, providing efficient unified memory utilization.

## Setup

```bash
brew install mlx-lm
mlx_lm.server --model mlx-community/gemma-2-9b-it-4bit
```

## Details

| Property | Value |
|----------|-------|
| Default port | 8080 |
| API type | OpenAI-compatible |
| VRAM reporting | No |
| Model format | MLX (safetensors) |
| Detection | `/version` endpoint or `lsof` process detection |

## Notes

- mlx-lm shares port 8080 with llama.cpp. asiai uses API probing and process detection to distinguish between them.
- Models use the HuggingFace/MLX community format (e.g., `mlx-community/gemma-2-9b-it-4bit`).
- Native MLX execution typically provides excellent performance on Apple Silicon.

## See also

Compare engines with `asiai bench --engines mlxlm` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/mtplx

Raw markdown: https://asiai.dev/markdown/engines/mtplx.md
Rendered: https://asiai.dev/engines/mtplx/

---
description: "MTPLX benchmark on Apple Silicon: MLX server with native MTP speculative decoding, OpenAI-compatible API."
---

# MTPLX

MTPLX is an MLX-based inference server for Apple Silicon ([youssofal/MTPLX](https://github.com/youssofal/MTPLX)) built around native MTP (multi-token prediction) speculative decoding. It exposes an OpenAI-compatible API and reports rich runtime state (generation mode, draft depth, session cache) on its `/health` endpoint.

## Setup

```bash
brew tap youssofal/mtplx
brew install mtplx
```

## Details

| Property | Value |
|----------|-------|
| Default port | Set at launch (`--port`) |
| API type | OpenAI-compatible |
| VRAM reporting | No (asiai measures GPU power/VRAM via ioreg) |
| Model format | MLX |
| Detection | `owned_by: "mtplx"` in `/v1/models`, version via `brew list --versions mtplx` |
| Requirements | Apple Silicon (M1+), macOS, Homebrew |

## Authentication

When the server is started with an API key, it requires `Authorization: Bearer <key>` on all routes — an unconfigured asiai then sees only 401s and cannot detect or monitor it. Point the engine's config entry at a file containing the key (the key itself never lives in the config):

```bash
asiai config add mtplx http://localhost:8080 --api-key-file ~/.config/asiai/keys/mtplx.key
```

asiai attaches the Bearer header to every request to this engine only (detection, monitoring, benchmarks). A missing or empty key file simply drops the header — no key material is ever logged. See [config](../commands/config.md#engine-api-keys---api-key-file) for details.

## Notes

- MTPLX has no fixed default port; asiai finds it on any scanned port through the `owned_by` field of `/v1/models`, and on non-standard ports through process discovery (`python -m mtplx.server.openai`).
- Its `/health` endpoint returns rich JSON (`{"ok": true, "generation_mode": "mtp", ...}`) rather than llama.cpp's `{"status": "ok"}`, so the two are never confused during detection.
- Generation runs through the standard streaming benchmark path; MTP speculative decoding is server-side and transparent to the OpenAI-compatible client.

## See also

Compare engines with `asiai bench --engines mtplx` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/ollama

Raw markdown: https://asiai.dev/markdown/engines/ollama.md
Rendered: https://asiai.dev/engines/ollama/

---
description: "How fast is Ollama on Apple Silicon? Benchmark setup, default port (11434), performance tips, and comparison with other engines."
---

# Ollama

Ollama is the most popular LLM inference engine for Mac, using a llama.cpp backend with GGUF models on port 11434. In our benchmarks on M4 Pro 64GB, it achieves 70 tok/s on Qwen3-Coder-30B but is 46% slower than LM Studio (MLX) for throughput.

[Ollama](https://ollama.com) is the most popular local LLM runner. asiai uses its native API.

## Setup

```bash
brew install ollama
ollama serve
ollama pull gemma2:9b
```

## Details

| Property | Value |
|----------|-------|
| Default port | 11434 |
| API type | Native (non-OpenAI) |
| VRAM reporting | Yes |
| Model format | GGUF |
| Load time measurement | Yes (via `/api/generate` cold start) |

## Notes

- Ollama reports VRAM usage per model, which asiai displays in benchmark and monitor output.
- Model names use the `name:tag` format (e.g., `gemma2:9b`, `qwen3.5:35b-a3b`).
- asiai sends `temperature: 0` for deterministic benchmark results.

## See also

See how Ollama compares: [Ollama vs LM Studio benchmark](../ollama-vs-lmstudio.md)



---

## /engines/omlx

Raw markdown: https://asiai.dev/markdown/engines/omlx.md
Rendered: https://asiai.dev/engines/omlx/

---
description: "oMLX benchmark on Apple Silicon: SSD KV caching, continuous batching, port 8000, and performance comparison."
---

# oMLX

oMLX is a native macOS inference server that uses paged SSD KV caching to handle larger context windows than memory alone would allow, with continuous batching for concurrent requests on port 8000. It supports both OpenAI and Anthropic-compatible APIs on Apple Silicon.

[oMLX](https://omlx.ai/) is a native macOS LLM inference server with paged SSD KV caching and continuous batching, managed from the menu bar. Built on MLX for Apple Silicon.

## Setup

```bash
brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
```

Or download the `.dmg` from [GitHub releases](https://github.com/jundot/omlx/releases).

## Details

| Property | Value |
|----------|-------|
| Default port | 8000 |
| API type | OpenAI-compatible + Anthropic-compatible |
| VRAM reporting | No |
| Model format | MLX (safetensors) |
| Detection | `/admin/info` JSON endpoint or `/admin` HTML page |
| Requirements | macOS 15+, Apple Silicon (M1+), 16 GB RAM min |

## Notes

- oMLX shares port 8000 with vllm-mlx. asiai uses `/admin/info` probing to distinguish between them.
- SSD KV caching enables larger context windows with lower memory pressure.
- Continuous batching improves throughput under concurrent requests.
- Supports text LLMs, vision-language models, OCR models, embeddings, and rerankers.
- The admin dashboard at `/admin` provides real-time server metrics.
- In-app auto-update when installed via `.dmg`.

## See also

Compare engines with `asiai bench --engines omlx` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/rapidmlx

Raw markdown: https://asiai.dev/markdown/engines/rapidmlx.md
Rendered: https://asiai.dev/engines/rapidmlx/

---
description: "Rapid-MLX benchmark on Apple Silicon: Homebrew packaging of vllm-mlx, OpenAI-compatible API, port 8000."
---

# Rapid-MLX

Rapid-MLX is a Homebrew packaging of the vllm-mlx engine (raullenchai upstream). The `rapid-mlx` wrapper delegates to the embedded vllm-mlx Python module in the formula's libexec virtualenv, so you get vllm-mlx's serving behaviour with a one-line `brew install` and no manual venv management.

## Setup

```bash
brew install raullenchai/rapid-mlx/rapid-mlx
# or, without Homebrew:
pip install rapid-mlx
```

## Details

| Property | Value |
|----------|-------|
| Default port | 8000 |
| API type | OpenAI-compatible |
| VRAM reporting | No (asiai measures GPU power/VRAM via ioreg) |
| Model format | MLX |
| Detection | `owned_by: "rapid-mlx"` in `/v1/models`, version via `rapid-mlx --version` |
| Requirements | Apple Silicon (M1+), macOS, Homebrew (for the brew install path) |

## Notes

- Rapid-MLX shares port 8000 with oMLX, vllm-mlx and vMLX. asiai disambiguates them by the `owned_by` field of `/v1/models` (`rapid-mlx`) and falls back to common brew install paths to resolve the binary.
- Because it wraps vllm-mlx, serving semantics (continuous batching, OpenAI-compatible endpoints) match vllm-mlx; the difference is purely packaging and update path (Homebrew vs pip/manual venv).
- `aisctl upgrade rapidmlx` is whitelisted (brew formula `raullenchai/rapid-mlx/rapid-mlx`) when asiai-inference-server is installed alongside.

## See also

Compare engines with `asiai bench --engines rapidmlx` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/vllm-mlx

Raw markdown: https://asiai.dev/markdown/engines/vllm-mlx.md
Rendered: https://asiai.dev/engines/vllm-mlx/

---
description: "vLLM-MLX on Apple Silicon: vLLM-compatible API on MLX, port 8000, Prometheus metrics, and benchmark data."
---

# vllm-mlx

vLLM-MLX brings the vLLM serving framework to Apple Silicon via MLX, offering continuous batching and an OpenAI-compatible API on port 8000. It can achieve 400+ tok/s on optimized models, making it one of the fastest options for concurrent inference on Mac.

[vllm-mlx](https://github.com/vllm-project/vllm) brings continuous batching to Apple Silicon via MLX.

## Setup

```bash
pip install vllm-mlx
vllm serve mlx-community/gemma-2-9b-it-4bit
```

## Details

| Property | Value |
|----------|-------|
| Default port | 8000 |
| API type | OpenAI-compatible |
| VRAM reporting | No |
| Model format | MLX (safetensors) |
| Detection | `/version` endpoint or `lsof` process detection |

## Notes

- vllm-mlx supports continuous batching, making it suitable for concurrent request handling.
- Can achieve 400+ tok/s on Apple Silicon with optimized models.
- Uses the standard vLLM OpenAI-compatible API.

## See also

Compare engines with `asiai bench --engines vllm-mlx` --- [learn how](../benchmark-llm-mac.md)



---

## /engines/vmlx

Raw markdown: https://asiai.dev/markdown/engines/vmlx.md
Rendered: https://asiai.dev/engines/vmlx/

---
description: "vMLX benchmark on Apple Silicon: MLX server with Mamba/SSM hybrid support, OpenAI-compatible API, port 8000."
---

# vMLX

vMLX is a high-performance MLX-based inference server with first-class support for Mamba/SSM hybrid architectures (DeltaNet, Mamba2, RetNet). It exposes an OpenAI-compatible API on port 8000 and identifies itself through a `/version` endpoint, with Prometheus metrics for inference activity.

[vMLX](https://vmlx.net/) targets Apple Silicon and is the only adapter here aimed at state-space / hybrid models alongside standard transformers.

## Setup

```bash
pip install vmlx
vmlx serve --model <repo-id-or-path> --port 8000
```

## Details

| Property | Value |
|----------|-------|
| Default port | 8000 |
| API type | OpenAI-compatible |
| VRAM reporting | No (asiai measures GPU power/VRAM via ioreg) |
| Model format | MLX |
| Detection | `/version` endpoint reporting `vmlx`, or `owned_by: "vmlx"` in `/v1/models` |
| Activity metrics | `/metrics` (Prometheus) |
| Requirements | Apple Silicon (M1+), macOS |

## Notes

- vMLX shares port 8000 with oMLX and vllm-mlx. asiai disambiguates them by probing `/version` and the `owned_by` field of `/v1/models`.
- First-class Mamba/SSM hybrid support (DeltaNet, Mamba2, RetNet) — useful for benchmarking non-transformer architectures that other MLX servers do not load.
- Version resolves from `/version`, falling back to `pip show vmlx`.

## See also

Compare engines with `asiai bench --engines vmlx` --- [learn how](../benchmark-llm-mac.md)



---

## /faq

Raw markdown: https://asiai.dev/markdown/faq.md
Rendered: https://asiai.dev/faq/

---
title: "Frequently Asked Questions"
description: "Common questions about asiai: supported engines, Apple Silicon requirements, benchmarking LLMs on Mac, RAM requirements, and more."
type: faq
faq:
  - q: "What is asiai?"
    a: "asiai is an open-source CLI tool that benchmarks and monitors LLM inference engines on Apple Silicon Macs. It supports 10 engines (Ollama, LM Studio, mlx-lm, llama.cpp, oMLX, vllm-mlx, vMLX, Rapid-MLX, MTPLX, Exo) and measures tok/s, TTFT, power consumption, and VRAM usage."
  - q: "What is the fastest LLM engine on Apple Silicon?"
    a: "In benchmarks on M4 Pro 64GB with Qwen3-Coder-30B, LM Studio (MLX backend) achieves 102 tok/s vs Ollama's 70 tok/s — 46% faster for token generation. However, Ollama has lower time-to-first-token latency."
  - q: "Does asiai work on Intel Macs?"
    a: "No. asiai requires Apple Silicon (M1, M2, M3, or M4). It uses macOS-specific APIs for GPU metrics, IOReport power monitoring, and hardware detection that are only available on Apple Silicon chips."
  - q: "How much RAM do I need to run LLMs locally?"
    a: "For a Q4-quantized 7B model: 8 GB minimum. For 13B: 16 GB. For 30B: 32-64 GB. MoE models like Qwen3.5-35B-A3B only use about 7 GB of active parameters, making them ideal for 16 GB Macs."
  - q: "Is Ollama or LM Studio better for Mac?"
    a: "It depends on your use case. LM Studio (MLX) is faster for throughput and more power-efficient. Ollama (llama.cpp) has lower first-token latency and handles large context windows (>32K) better. See the detailed comparison at asiai.dev/ollama-vs-lmstudio."
  - q: "Does asiai require sudo or root access?"
    a: "No. All features including GPU observability (ioreg) and power monitoring (IOReport) work without sudo. The optional --power flag for cross-validation with powermetrics is the only feature that uses sudo."
  - q: "How do I install asiai?"
    a: "Install via pip (pip install asiai) or Homebrew (brew tap druide67/tap && brew install asiai). Python 3.11+ required."
  - q: "Can AI agents use asiai?"
    a: "Yes. asiai includes an MCP server with 14 tools and 3 resources. Install with pip install asiai[mcp] and configure as asiai mcp in your MCP client (Claude Code, Cursor, etc.)."
  - q: "How accurate are the power measurements?"
    a: "IOReport power readings have less than 1.5% delta compared to sudo powermetrics, validated across 20 samples on both LM Studio (MLX) and Ollama (llama.cpp)."
  - q: "Can I benchmark multiple models at once?"
    a: "Yes. Use asiai bench --compare to run cross-model benchmarks. Supports model@engine syntax for precise control, with up to 8 comparison slots."
  - q: "How do I share my benchmark results?"
    a: "Run asiai bench --share to anonymously submit results to the community leaderboard. Add --card to generate a shareable 1200x630 benchmark card image."
  - q: "What metrics does asiai measure?"
    a: "Seven core metrics: tok/s (generation speed), TTFT (time to first token), power (GPU+CPU watts), tok/s/W (energy efficiency), VRAM usage, run-to-run stability, and thermal throttling state."
---

# Frequently Asked Questions

## General

**What is asiai?**

asiai is an open-source CLI tool that benchmarks and monitors LLM inference engines on Apple Silicon Macs. It supports 10 engines (Ollama, LM Studio, mlx-lm, llama.cpp, oMLX, vllm-mlx, vMLX, Rapid-MLX, MTPLX, Exo) and measures tok/s, TTFT, power consumption, and VRAM usage with zero dependencies.

**Does asiai work on Intel Macs or Linux?**

No. asiai requires Apple Silicon (M1, M2, M3, or M4). It uses macOS-specific APIs (`sysctl`, `vm_stat`, `ioreg`, `IOReport`, `launchd`) that are only available on Apple Silicon Macs.

**Does asiai require sudo or root access?**

No. All features including GPU observability (`ioreg`) and power monitoring (`IOReport`) work without sudo. The optional `--power` flag for cross-validation with `powermetrics` is the only feature that uses sudo.

## Engines & Performance

**What is the fastest LLM engine on Apple Silicon?**

In our benchmarks on M4 Pro 64GB with Qwen3-Coder-30B (Q4_K_M), LM Studio (MLX backend) achieves **102 tok/s** vs Ollama's **70 tok/s** — 46% faster for token generation. LM Studio is also 82% more power-efficient (8.23 vs 4.53 tok/s/W). See our [detailed comparison](ollama-vs-lmstudio.md).

**Is Ollama or LM Studio better for Mac?**

It depends on your use case:

- **LM Studio (MLX)**: Best for throughput (code generation, long responses). Faster, more efficient, lower VRAM.
- **Ollama (llama.cpp)**: Best for latency (chatbots, interactive use). Faster TTFT. Better for large context windows (>32K tokens).

**How much RAM do I need to run LLMs locally?**

| Model Size | Quantization | RAM Needed |
|-----------|-------------|-----------|
| 7B | Q4_K_M | 8 GB minimum |
| 13B | Q4_K_M | 16 GB minimum |
| 30B | Q4_K_M | 32-64 GB |
| 35B MoE (3B active) | Q4_K_M | 16 GB (only active params loaded) |

## Benchmarking

**How do I run my first benchmark?**

Three commands:

```bash
pip install asiai     # Install
asiai detect          # Find engines
asiai bench           # Run benchmark
```

**How long does a benchmark take?**

A quick benchmark (`asiai bench --quick`) takes about 2 minutes. A full cross-engine comparison with multiple prompts and 3 runs takes 10-15 minutes.

**How accurate are the power measurements?**

IOReport power readings have less than 1.5% delta compared to `sudo powermetrics`, validated across 20 samples on both LM Studio (MLX) and Ollama (llama.cpp).

**Can I compare my results with other Mac users?**

Yes. Run `asiai bench --share` to anonymously submit results to the [community leaderboard](leaderboard.md). Use `asiai compare` to see how your Mac stacks up.

## AI Agent Integration

**Can AI agents use asiai?**

Yes. asiai includes an MCP server with 14 tools and 3 resources. Install with `pip install "asiai[mcp]"` and configure as `asiai mcp` in your MCP client (Claude Code, Cursor, Windsurf). See the [Agent Integration Guide](agent.md).

**What MCP tools are available?**

14 tools: `check_inference_health`, `get_inference_snapshot`, `list_models`, `detect_engines`, `run_benchmark`, `get_recommendations`, `diagnose`, `get_metrics_history`, `get_benchmark_history`, `refresh_engines`, `compare_engines`, `get_fleet_snapshot`, `get_fleet_health`, `fleet_audit_tail`.

3 resources: `asiai://status`, `asiai://models`, `asiai://system`.



---

## /fleet-mode

Raw markdown: https://asiai.dev/markdown/fleet-mode.md
Rendered: https://asiai.dev/fleet-mode/

# Fleet mode — multi-host observability + cross-host writes

`asiai` started as a single-host observability and benchmarking tool.
**Fleet mode** lets you declare several hosts that all run `asiai web`
and view their state side-by-side from one machine — without re-running
`asiai monitor` on every Mac you own.

- **Phase 1 — read-only** (shipped in `asiai`): list nodes, poll each
  one's snapshot in parallel, render a CLI table and an HTML grid.
- **Phase 2 — writes** (shipped in `asiai-inference-server` + Bearer
  auth in `asiai`): execute `purge`, `stop/start/restart`, `unload`,
  `install/uninstall`, `upgrade` on remote nodes via authenticated
  HTTP. See the **Phase 2 — write commands** section below.
- **Phase 3 — auto-discovery** (planned): mDNS Bonjour
  `_asiai-fleet._tcp.local`, TUI fleet panel, TLS/mTLS for off-LAN.

## What you need

- One Mac per node, each running `asiai >= 1.4` with the `web` extra
  installed (`pip install asiai[web]` or `pip install asiai[all]`).
- `asiai web` running on each remote node, **bound on a network
  interface that the orchestrator host can reach**:

  ```sh
  asiai web --host 0.0.0.0 --port 8899
  ```

  ⚠️ Binding `0.0.0.0` exposes the dashboard to every host on the local
  network. By default `asiai web` binds `127.0.0.1` and warns if you
  opt-in to `0.0.0.0`. Use Phase 1 only on a trusted LAN.

- On the orchestrator host, just `asiai` installed. The orchestrator
  does not need to run `asiai web` itself — it only needs CLI access.

## Configure the fleet

The fleet config lives at `~/.config/asiai/fleet.json` (perms `0o600`).
It is hand-editable but the CLI manages it for you:

```sh
asiai fleet add studio    --url http://192.0.2.10:8899 --role workstation
asiai fleet add laptop    --url http://192.0.2.11:8899 --role spare
asiai fleet add minihost  --url http://minihost.local:8899
asiai fleet list
```

`--role` is free text; use it for whatever taxonomy fits your setup.
The URL is passed verbatim to the HTTP client, so any hostname, IP, or
mDNS `.local` name works — including Tailscale `*.ts.net` addresses.

## Poll node status

`asiai fleet status` polls every node's `GET /api/v1/snapshot`
endpoint in parallel and prints an aggregated table:

```sh
asiai fleet status
asiai fleet status --json | jq '.nodes[] | {nickname, ok, latency_ms}'
asiai fleet ping studio        # check one node only
```

Per-node timeout is 5 seconds by default (`--timeout`). The aggregate
poll has a defensive 10-second cap: if a node's TCP connection dies
silently, the rest of the fleet's status is still returned.

The exit code is `0` when every node responded successfully, and `1` if
at least one node is down — convenient for shell scripts and CI checks.

## Browse the fleet in the web dashboard

`asiai web` on the orchestrator host gains a new `/fleet` page that
shows a card per node, refreshed every 10 seconds via HTMX. Each card
displays the engine list reported by the remote, the request latency,
and a status badge.

```sh
asiai web                          # main host on http://127.0.0.1:8899
open http://127.0.0.1:8899/fleet
```

## Phase 2 — write commands

Phase 2 adds **authenticated cross-host writes**. From the orchestrator,
`aisctl fleet push <nickname> <command>` POSTs a command to the
remote node's `asiai web`, which validates the Bearer token, applies a
per-token rate limit, writes an audit log line, and proxies to a
loopback `aisctl serve` companion process that runs the actual command.

Why two processes on the node? `asiai web` is the only LAN-facing
surface, so the trust boundary stays in one place. `aisctl serve`
listens on `127.0.0.1:8898` only and shares a per-startup loopback
secret with `asiai web` (file `~/.local/state/asiai/aisctl-serve-token`,
0o600). A LAN attacker who somehow bypasses `asiai web`'s auth still
needs filesystem access to that loopback token to reach `aisctl serve`.

### Whitelisted commands

| Command | Args required | Upstream timeout | Notes |
|---------|---------------|------------------|-------|
| `purge` | — | 30 s | `sudo /usr/sbin/purge` (low risk) |
| `unload <engine> [<model>]` | `engine` | 60 s | Native API unload, fallback restart |
| `stop <engine>` | `engine` | 60 s | LaunchDaemon `bootout` |
| `start <engine>` | `engine` | 120 s | LaunchDaemon `bootstrap` + health |
| `restart <engine>` | `engine` | 120 s | stop + start |
| `install <engine>` | `engine` | 300 s | Provision plist + sudoers |
| `uninstall <engine>` | `engine` | 120 s | Remove plist + pf anchor |
| `upgrade <engine>` | `engine` | 600 s | `brew upgrade` (formulas whitelisted) |

Anything outside this whitelist is rejected at the LAN edge with HTTP
400 before any subprocess is spawned. `upgrade` additionally enforces
a per-engine Homebrew formula whitelist (`ollama`, `llama.cpp`,
`lm-studio`, `rapid-mlx`, `turboquant`) to defend against argv
injection even if the engine regex is bypassed.

### Bootstrapping the auth surface on a node

```sh
# 1. On the node — initialize the auth file and copy the secret ONCE.
asiai auth init
# → prints token id + secret. Save the secret — asiai will never show it again.

# 2. On the node — start the loopback companion (one-shot or LaunchDaemon).
aisctl serve &
# → listens on 127.0.0.1:8898, writes ~/.local/state/asiai/aisctl-serve-token

# 3. On the orchestrator — register the node WITH the secret.
asiai fleet add studio \
  --url http://192.0.2.10:8899 \
  --role workstation \
  --auth-token <the-secret-from-step-1>
```

### Issuing a write

```sh
# Free unified memory on the studio.
aisctl fleet push studio purge

# Restart Ollama everywhere (loop in shell — no native broadcast yet).
for n in studio laptop minihost; do
  aisctl fleet push "$n" restart --engine ollama
done

# Unload a specific Ollama model without restarting the daemon.
aisctl fleet push studio unload --engine ollama --model llama3.2
```

`aisctl fleet push --json` emits a single JSON object per call so
agents and CI pipelines can parse the result.

### Token lifecycle on the node

```sh
asiai auth list                       # list tokens (no secrets shown)
asiai auth create --label laptop-2026  # add a new token
asiai auth rotate tok_abc123def456     # revoke + replace (returns new secret)
asiai auth revoke tok_abc123def456     # revoke without replacing
```

### Audit log

Every write attempt (denied or executed) appends one JSON object to
`~/.local/share/asiai/fleet-audit.jsonl` (0o600, rotated at 10 MB to
`.1`). Fields: `ts`, `source_ip`, `token_id`, `nickname`, `command`,
`args` (with secret-bearing keys redacted), `status`, `http_status`,
`duration_ms`, `exit_code`, `error`. Useful both for forensics and
for confirming that a command actually ran on the right host.

Four ways to read it, by audience:

- **Humans** — the dashboard's Journal page and the cockpit drawer
  (operator session required).
- **Operators at a terminal** — `asiai fleet audit` on the host that
  owns the file (filters: `--limit`, `--actor`, `--status`, `--since`;
  `--json` for raw events). A direct local read, no session needed.
- **Machines on the hub** — `GET /api/v1/fleet/audit` (raw events,
  full operator session required).
- **AI agents** — the redacted one-shot exchange below.

### Operator login scopes

`asiai auth login` mints a single-use operator code. The scope is
**bound to the code at mint** and cannot be widened at exchange:

```sh
asiai auth login                      # scope "full" — opens a dashboard write session
asiai auth login --scope audit:read   # buys exactly ONE redacted audit read, nothing else
```

Each scope has exactly one exchange surface. A `full` code is only
accepted by the `/login` form; an `audit:read` code is only accepted by
the one-shot exchange route — each is refused (and burned) by the other
surface, so a code can never do more than what it was minted for.

### Audit access for agents (MCP)

An LLM agent asking "what happened on the fleet tonight?" is a
legitimate use case — but the agent's context may leave the machine, so
the journal must never reach it raw. The `fleet_audit_tail` MCP tool
(and the underlying `POST /api/v1/fleet/audit-tail` route) implements a
hardened path:

```sh
# 1. Mint a read-scoped code in a trusted shell:
asiai auth login --scope audit:read

# 2. The agent exchanges it for ONE redacted read:
#    MCP: fleet_audit_tail(code="aop_...", lines=50, since_hours=6)
#    or:  POST /api/v1/fleet/audit-tail  {"code": "aop_...", "lines": 50}
```

Guarantees, enforced server-side:

- **One code, one read** — the code is consumed by the exchange; no
  session is created, so there is nothing to revoke.
- **Metadata only** — a strict field whitelist (`ts`, actor, verb,
  target, status, durations). Raw command `args` and free-form `error`
  text never pass; neither do codes, tokens or secrets.
- **Bounded** — `lines` ≤ 200, `since_hours` ≤ 24, all-requests rate
  limit on the route.
- **Self-journaled** — every read lands in the journal as an
  `audit_read` event with an exchange id.

What this is NOT: an access-control barrier against local agents (any
process with shell access can mint a code itself). It buys attribution,
short TTLs and a single audited read path — the honest rationale and
the deployment trade-offs (local vs cloud LLM) are recorded in
[ADR 0001](adr/0001-audit-journal-read-for-local-agents.md).

### What is NOT in Phase 2

- **Auto-discovery** (Bonjour/mDNS) — Phase 3.
- **TLS** between the orchestrator and the nodes — Phase 3.
- **TUI fleet panel** — Phase 3.
- **Broadcast commands** ("purge everywhere in one call") — would land
  before Phase 3 if the use case appears in practice; the shell loop
  above is fine for small fleets.
- **MCP write tools** — Phase 3.

## Security notes

### Phase 1 (read-only) defences

1. The Phase 1 dashboard is **read-only**: a compromised LAN peer can
   read your engine list and monitoring metrics but cannot trigger an
   inference run, purge memory, or restart an engine. This limits the
   blast radius significantly.
2. `fleet.json` is saved with `0o600` perms so other accounts on the
   same Mac cannot enumerate which hosts you monitor.
3. `auth_token` is never echoed by the JSON API or the HTML page
   (verified by the test suite).
4. If you need to expose nodes off-LAN, put them behind a VPN
   (Tailscale, WireGuard) rather than punching firewall holes. The
   `asiai_url` accepts the VPN hostname directly.

### Phase 2 (writes) threat model

| Threat | Defence |
|--------|---------|
| Unauthenticated LAN peer issues writes | `asiai web` requires `Authorization: Bearer <secret>`; missing → 401. |
| Brute-force secret enumeration | `secrets.token_urlsafe(32)` = 256 bits of entropy; constant-time `hmac.compare_digest` comparison against salted SHA-256 hashes; per-token rate limit (30/min). |
| Stolen secret used from elsewhere | `asiai auth rotate <id>` revokes + replaces in one step; audit log captures source IP per call. |
| Command injection through engine/model args | LAN edge regex (`^[a-z][a-z0-9_-]{0,31}$` for engines, HF naming for models); `subprocess.run(list)` without `shell=True`; `upgrade` uses a per-engine Homebrew formula whitelist. |
| Other local user POSTs directly to `aisctl serve` to bypass `asiai web`'s checks | `aisctl serve` binds 127.0.0.1 only AND requires a per-startup loopback Bearer secret stored at `~/.local/state/asiai/aisctl-serve-token` (0o600). |
| Token leaked in CLI output / shell history | Plaintext secret is shown EXACTLY ONCE at create/rotate time; the on-disk hash cannot be reversed. |
| Replay after a destructive command | Audit log JSONL keeps `ts`, `source_ip`, `token_id`, command, exit code per attempt (rotated at 10 MB). |
| Body-size DoS on the auth endpoint | LAN edge caps the request body at 64 KB; oversize → 413. |
| Symlink swap on auth.json or fleet.json | `save_*` refuses to write through a symlink. |
| Concurrent token CRUD corrupting auth.json | `fcntl.flock` cross-process lock around every read-modify-write. |

### Limits that are explicit non-goals for Phase 2

- **No TLS** between orchestrator and nodes. Phase 2 is for trusted
  LANs (or LANs glued together by a VPN). If you need confidentiality
  on the wire, route through Tailscale / WireGuard / SSH tunnel.
- **No mTLS.** Token-only auth.
- **No multi-user RBAC.** Every token has the same capabilities — the
  whole whitelist. Splitting into "read-only" vs "operator" tokens is
  a Phase 3 candidate if the use case emerges.
- **No off-host audit shipping.** The JSONL stays on the node; ship
  it elsewhere with `fluent-bit`/`vector` if you need centralized
  storage. Rotation discards the previous backup after 10 MB.

## Backup & restore

The fleet config is a single file: `~/.config/asiai/fleet.json` (0o600).
Back it up with a regular file copy:

```sh
cp ~/.config/asiai/fleet.json ~/Documents/asiai-fleet-backup.json
```

To move a fleet declaration to another Mac, copy the same file across
and `chmod 600` it on the target.

## How does this compare to LM Link / Ollama / Exo?

asiai fleet is **engine-agnostic observability** for an Apple Silicon
home lab. It sits at a different layer than the alternatives:

- **LM Studio "LM Link"** (Tailscale-based, since Feb 2026): makes one
  LM Studio model on a remote Mac reachable via local `localhost:1234`.
  Solves "use my Mac Studio's model from my MacBook". Single engine,
  single model at a time. asiai fleet shows you what is running on N
  Macs across **all engines simultaneously**.
- **Ollama**: no native multi-host mode in 2026. Third-party
  load-balancers (e.g. OLOL) exist for Ollama clusters specifically.
- **Exo** (exo-explore/exo): distributed inference — shards one large
  model across N Macs. Different layer entirely; complementary, not
  competing.

The asiai differentiator: **see Ollama + LM Studio + mlx-lm +
llama.cpp + Rapid-MLX/vllm-mlx side-by-side across your Macs**, with
energy and thermal observability via IOReport (Apple Silicon-only,
unique among comparable tools as of May 2026).

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `status: DOWN error: ConnectionRefusedError` | Remote `asiai web` not running | `asiai web --host 0.0.0.0` on the remote |
| `status: DOWN error: TimeoutError` | Network reachability or firewall | `curl -v http://<host>:8899/api/v1/status` from the orchestrator |
| `status: DOWN error: HTTP 404` | Remote runs an older `asiai` without the `/api/v1/snapshot` endpoint | Upgrade the remote to the same `asiai` version as the orchestrator |
| All nodes "never" last seen | `fleet status` has never run since the nodes were added | Run `asiai fleet status` once to populate the `last_seen` field |



---

## /getting-started

Raw markdown: https://asiai.dev/markdown/getting-started.md
Rendered: https://asiai.dev/getting-started/

---
description: Install asiai and run your first LLM benchmark in under 2 minutes. One command, zero dependencies, works on any Apple Silicon Mac.
---

# Getting Started

**Apple Silicon AI** — Multi-engine LLM benchmark & monitoring CLI.

asiai compares inference engines side-by-side on your Mac. Load the same model on Ollama and LM Studio, run `asiai bench`, get the numbers. No guessing, no vibes — just tok/s, TTFT, power efficiency, and stability per engine.

## Quick start

```bash
pipx install asiai        # Recommended: isolated install
```

Or via Homebrew:

```bash
brew tap druide67/tap
brew install asiai
```

Other options:

```bash
uvx asiai detect           # Run without installing (requires uv)
pip install asiai           # Standard pip install
```

### First launch

```bash
asiai setup                # Interactive wizard — detects hardware, engines, models
asiai detect               # Or jump straight to engine detection
```

Then benchmark:

```bash
asiai bench -m qwen3.5 --runs 3 --power
```

Example output:

```
  Mac Mini M4 Pro — Apple M4 Pro  RAM: 64.0 GB (42% used)  Pressure: normal

Benchmark: qwen3.5

  Engine       tok/s (±stddev)    Tokens   Duration     TTFT       VRAM    Thermal
  ────────── ───────────────── ───────── ────────── ──────── ────────── ──────────
  lmstudio    72.6 ± 0.0 (stable)   435    6.20s    0.28s        —    nominal
  ollama      30.4 ± 0.1 (stable)   448   15.28s    0.25s   26.0 GB   nominal

  Winner: lmstudio (2.4x faster)
  Power: lmstudio 13.2W (5.52 tok/s/W) — ollama 16.0W (1.89 tok/s/W)
```

## What it measures

| Metric | Description |
|--------|-------------|
| **tok/s** | Generation speed (tokens/sec), excluding prompt processing |
| **TTFT** | Time to first token — prompt processing latency |
| **Power** | GPU power draw in watts (`sudo powermetrics`) |
| **tok/s/W** | Energy efficiency — tokens per second per watt |
| **Stability** | Run-to-run variance: stable (<5%), variable (<10%), unstable (>10%) |
| **VRAM** | Memory footprint — native (Ollama, LM Studio) or estimated via `ri_phys_footprint` (all engines) |
| **Thermal** | CPU throttling state and speed limit percentage |

## Supported engines

| Engine | Port | API |
|--------|------|-----|
| [Ollama](https://ollama.com) | 11434 | Native |
| [LM Studio](https://lmstudio.ai) | 1234 | OpenAI-compatible |
| [mlx-lm](https://github.com/ml-explore/mlx-examples) | 8080 | OpenAI-compatible |
| [llama.cpp](https://github.com/ggml-org/llama.cpp) | 8080 | OpenAI-compatible |
| [oMLX](https://github.com/jundot/omlx) | 8000 | OpenAI-compatible |
| [vllm-mlx](https://github.com/vllm-project/vllm) | 8000 | OpenAI-compatible |
| [Exo](https://github.com/exo-explore/exo) | 52415 | OpenAI-compatible |

## Custom ports

If your engine runs on a non-standard port, asiai will usually find it automatically via process detection. You can also register it manually:

```bash
asiai config add omlx http://localhost:8800 --label desktop
```

Manually added engines are persisted and never auto-pruned. See [config](commands/config.md) for details.

## Requirements

- macOS on Apple Silicon (M1 / M2 / M3 / M4)
- Python 3.11+
- At least one inference engine running locally

## Zero dependencies

The core uses only the Python standard library — `urllib`, `sqlite3`, `subprocess`, `argparse`. No `requests`, no `psutil`, no `rich`.

Optional extras:

- `asiai[web]` — FastAPI web dashboard with charts
- `asiai[tui]` — Textual terminal dashboard
- `asiai[mcp]` — MCP server for AI agent integration
- `asiai[all]` — Web + TUI + MCP
- `asiai[dev]` — pytest, ruff, pytest-cov



---

## /index

Raw markdown: https://asiai.dev/markdown/index.md
Rendered: https://asiai.dev/index/

---
template: home.html
title: asiai — Multi-engine LLM benchmark for Apple Silicon
description: Compare LLM inference engines on Apple Silicon. Measure tok/s, TTFT, GPU power and thermal across 10 engines. Open source CLI.
hide:
  - navigation
  - toc
---



---

## /installation

Raw markdown: https://asiai.dev/markdown/installation.md
Rendered: https://asiai.dev/installation/

---
description: Install asiai via pip, Homebrew, or from source. Requirements: macOS on Apple Silicon (M1+), Python 3.11+.
---

# Installation

## pipx (recommended)

```bash
pipx install asiai
```

## Homebrew

```bash
brew tap druide67/tap
brew install asiai
```

## pip

```bash
pip install asiai
```

## Try without installing

```bash
uvx asiai detect           # Requires uv
```

## Optional extras

```bash
pip install "asiai[web]"   # Web dashboard (FastAPI + charts)
pip install "asiai[tui]"   # Terminal dashboard (Textual)
pip install "asiai[mcp]"   # MCP server for AI agents
pip install "asiai[all]"   # Web + TUI + MCP
```

## From source

```bash
git clone https://github.com/druide67/asiai.git
cd asiai
pip install -e ".[dev]"
```

## Verify installation

```bash
asiai --version
asiai setup                # Interactive wizard
asiai detect               # Or detect engines directly
```



---

## /known-issues

Raw markdown: https://asiai.dev/markdown/known-issues.md
Rendered: https://asiai.dev/known-issues/

# Known issues

Tracked defects that are deliberately deferred (low impact, or a fix carries
more risk than the bug). Each entry says why it's open and what a fix would take.

## ioreport: CFString leak in `_unwrap_to_array` (low)

`asiai/collectors/ioreport.py` calls `_cfstr("IOReportChannels")` on every
`IOReportSampler.sample()` (via `_unwrap_to_array`). `CFStringCreateWithCString`
returns an owned reference (CoreFoundation Create Rule) that is never released,
so each sample leaks one small CFString (~tens of bytes). Pre-existing, not
introduced by the 1.11.0 overhaul.

- **Impact**: a slow leak over a long bench (thousands of samples). Negligible
  for normal runs; matters only for a very long-lived monitor process.
- **Why deferred**: the fix needs a `CFRelease` ctypes binding, and releasing the
  wrong object segfaults the process. Not worth that risk inside the 1.11.0
  metrics work.
- **Fix sketch**: bind `CFRelease` (`argtypes=[c_void_p]`), cache the
  `"IOReportChannels"` CFString once at module load instead of recreating it per
  call, and release any CFStrings created per sample. Cover with a soak test.

## agentic: `_compute_reuse` and `_compute_verdict` use different early-stop filters (low)

`asiai/benchmark/agentic.py`: the legacy categorical `_compute_verdict` filters
runs on `error is None` only, while `_compute_reuse` additionally excludes
early-stopped runs. On a run set containing early-stops the published
`prefix_cache_reuse_verdict` (string) and the `prefix_cache_reuse.reuse_fraction`
can therefore be computed over slightly different run subsets.

- **Impact**: cosmetic. The verdict is explicitly tagged engine-family-specific
  and consumers are told to use the raw `reuse_fraction` signal, not the verdict.
- **Why deferred**: aligning the filters changes the legacy verdict's value on
  early-stop runs; no real decision depends on it.
- **Fix sketch**: have `_compute_verdict` reuse the same early-stop exclusion as
  `_compute_reuse` (factor the `_ok(run)` predicate out and share it).



---

## /leaderboard

Raw markdown: https://asiai.dev/markdown/leaderboard.md
Rendered: https://asiai.dev/leaderboard/

---
description: Community benchmark results across Apple Silicon Macs. Compare tok/s by engine, model and hardware. Submit your own results.
---

# Community Leaderboard

<div id="leaderboard-app">

<div class="lb-filters" markdown>
<div class="lb-filter-row">
  <input type="text" id="lb-chip" placeholder="Filter by chip (e.g. M4 Pro)" class="lb-input">
  <input type="text" id="lb-model" placeholder="Filter by model (e.g. qwen2.5)" class="lb-input">
  <button id="lb-search" class="lb-btn">Search</button>
  <button id="lb-clear" class="lb-btn lb-btn-secondary">Clear</button>
</div>
<div class="lb-status">
  <span id="lb-count"></span>
  <span id="lb-updated"></span>
</div>
</div>

<div class="lb-table-wrap">
<table id="lb-table" class="lb-table">
<thead>
<tr>
  <th class="lb-sortable" data-col="engine">Engine</th>
  <th class="lb-sortable" data-col="model">Model</th>
  <th class="lb-sortable lb-active-sort" data-col="median_tok_s">tok/s</th>
  <th class="lb-sortable" data-col="median_ttft_ms">TTFT</th>
  <th>Chip · RAM</th>
  <th>Quant</th>
  <th class="lb-sortable" data-col="median_power_watts">W</th>
  <th class="lb-sortable" data-col="median_tok_s_per_watt">tok/s/W</th>
  <th class="lb-sortable" data-col="last_submitted_at">Last seen</th>
  <th class="lb-sortable" data-col="samples">Samples</th>
</tr>
</thead>
<tbody id="lb-body">
<tr><td colspan="10" class="lb-loading">Loading community data...</td></tr>
</tbody>
</table>
</div>

<div class="lb-footer">
  <p>Data from <a href="https://api.asiai.dev/api/v1/leaderboard">api.asiai.dev</a> — updated live.<br>
  Contribute your results: <code>asiai bench --share</code></p>
</div>

</div>



---

## /methodology

Raw markdown: https://asiai.dev/markdown/methodology.md
Rendered: https://asiai.dev/methodology/

---
description: How asiai measures tok/s, TTFT and power. Warmup, statistical methodology, and why results are reproducible.
---

# Benchmark Methodology

asiai follows established benchmarking standards ([MLPerf](https://mlcommons.org/benchmarks/inference-server/), [SPEC CPU 2017](https://www.spec.org/cpu2017/), [NVIDIA GenAI-Perf](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/benchmarking/genai_perf.html)) to produce reliable, reproducible, and comparable results.

## Protocol

1. **Pre-flight gate check**: Refuse to start if memory pressure is critical or system is heavily throttled (<80%)
2. **Warmup**: 1 non-timed generation per engine to prime JIT compilers and caches
3. **Measured runs**: Default 3 runs per prompt per engine (configurable via `--runs`)
4. **Sampling**: `temperature=0` (greedy) for deterministic output
5. **Model unloading**: After benchmarking each engine, the model is unloaded to free unified memory before the next engine starts. This prevents memory accumulation and swapping when comparing multiple engines on large models
6. **Adaptive cooldown**: After unloading, asiai waits for macOS memory pressure to return to "normal" (max 30s), then adds a minimum 5s thermal cooldown
7. **Sanity checks**: Results with tok/s ≤ 0 are discarded. TTFT > 60s or tok/s > 500 trigger warnings (likely swapping or measurement errors)
8. **Reporting**: Median tok/s as primary metric (SPEC standard), mean ± stddev as secondary
9. **Throttling**: Warning emitted if `thermal_speed_limit < 100%` during any run. Thermal drift (monotone tok/s decrease across runs, ≥5% drop) is detected and reported
10. **Metadata**: Engine version, model format, quantization, hardware chip, macOS version stored per result

## Metrics

### tok/s — Generation Speed

Tokens per second of **generation time only**, excluding prompt processing (TTFT).

**Ollama** (native API, `/api/generate`):
```
tok_per_sec = eval_count / (eval_duration_ns / 1e9)
```
Source: internal GPU timing reported by Ollama. No network overhead. This is the most accurate measurement.

**OpenAI-compatible engines** (LM Studio, llama.cpp, mlx-lm, vllm-mlx):
```
generation_s = wall_clock_s - ttft_s
tok_per_sec  = completion_tokens / generation_s
```
Source: client-side wall clock via streaming SSE. Includes HTTP overhead per chunk (~1% slower than server-side timing, validated by cross-validation).

**Token count**: from `usage.completion_tokens` in the server response. If the server does not report this field, asiai falls back to `len(text) // 4` and logs a warning. This fallback can be ~25% off.

**Cross-validation** (April 2026, Qwen3.5-35B NVFP4, M4 Pro 64GB):

| Method | tok/s | Delta vs reference |
|--------|-------|--------------------|
| Ollama native (internal GPU) | 66.6 | reference |
| OpenAI streaming (client) | 66.1 | -0.8% |

At large context sizes (e.g., 64k tokens), TTFT can dominate total duration. Excluding it from tok/s prevents fast generators from appearing slow.

### TTFT — Time to First Token

Time between sending the request and receiving the first output token, in milliseconds.

Since v1.6.0, asiai measures **two TTFT values** for Ollama, and one for all other engines:

**Ollama** (dual measurement):

- **Server-side TTFT** (`ttft_ms`): extracted from `prompt_eval_duration` in the Ollama response. This is pure GPU prompt processing time with zero network overhead — the most accurate measurement possible. Reported as `ttft_source: server`.
- **Client-side TTFT** (`ttft_client_ms`): measured at the arrival of the first SSE content chunk. Includes HTTP setup, request transmission, and server processing. This is the same method used for all other engines.

**OpenAI-compatible engines** (LM Studio, llama.cpp, mlx-lm, vllm-mlx):

- **Client-side TTFT** (`ttft_client_ms`): measured at the first SSE content chunk. This is the only measurement available since these engines do not expose internal prompt processing timing. Both `ttft_ms` and `ttft_client_ms` contain the same value.

**Comparable metric**: `ttft_client_ms` is the **cross-engine comparable** metric — it uses the same measurement method regardless of the engine. Use this when comparing TTFT across different engines. The server-side `ttft_ms` from Ollama is more accurate for absolute prompt processing time, but not directly comparable with other engines.

**Cross-validation** (April 2026, Qwen3.5-35B NVFP4, M4 Pro 64GB):

| Method | TTFT | Delta |
|--------|------|-------|
| Ollama server-side (`ttft_ms`) | 27 ms | reference |
| Ollama client-side (`ttft_client_ms`) | 51 ms | +24 ms |

The 24ms delta represents HTTP overhead on localhost. This overhead is consistent and predictable but significant enough to matter when comparing engines.

### Power — GPU Watts

Average GPU power during execution, measured via Apple's IOReport Energy Model framework (no sudo required). One measurement per engine — not session-wide averaging.

### tok/s/W — Energy Efficiency

```
tok_per_sec_per_watt = tok_per_sec / power_watts
```

### Variance — Pooled Stddev

Pooled intra-prompt standard deviation captures run-to-run noise **without** mixing in inter-prompt variance. Uses Bessel's correction (N-1 denominator) for unbiased sample variance.

Stability classification:

- CV < 5% → `stable`
- CV < 10% → `variable`
- CV >= 10% → `unstable`

Where CV = `(std_dev / mean) * 100`.

### VRAM — Memory Usage

**Primary**: engine-native API (Ollama `/api/ps`, LM Studio `/v1/models`).
**Fallback**: `ri_phys_footprint` via ctypes (same as Activity Monitor). Marked "(est.)" in the UI.

## Agentic Mode — Prefix Cache Reuse Benchmark

Standard single-shot benchmarks measure how fast an engine generates tokens
in isolation. They miss the dominant cost pattern of multi-turn agent
workloads: a long shared **system** prompt (tools, rules, persona — often
6K+ tokens) plus a short **user** message that changes every turn. An
engine that does not reuse the cached prefix re-processes those 6K tokens
on every call, and TTFT explodes.

`asiai bench --agentic-mode` runs an 8-phase protocol designed to expose
this behavior explicitly.

### Protocol

| Phase | System | User | max_tokens | Purpose |
|---|---|---|---|---|
| `cold` | SYS_A | USER_X | 400 | First run, no cache |
| `warm` | SYS_A | USER_X | 400 | Same request — full cache hit |
| `prefix-test-1` | SYS_A | USER_Y | 400 | **Sys identical, user different** — the real test |
| `prefix-test-2` | SYS_A | USER_X | 400 | Back to USER_X — should be cache hit |
| `prefix-test-3` | SYS_A | USER_Y | 400 | Repeat the cross-user pattern |
| `cold-prefix` | SYS_B | USER_X | 400 | Sys changes — should miss cache |
| `long-context` | SYS_A | USER_L (~50K tok) | 200 | Saturate decode at long context |
| `long-prefix` | SYS_A | USER_L | 200 | Same long context — cache hit |

Prompts are generated deterministically with a sentinel pattern that
breaks naive substring caches; sizes are calibrated for Qwen-family
tokenizers (~5.3 chars/token on English prose).

### Verdict

`prefix_cache_reuse` is computed from the prefix-test phases:

1. **Primary signal** — if the engine reports `usage.prompt_tokens_details.cached_tokens`
   in its streaming response (llama.cpp, mlx-lm), the ratio `cached / prompt`
   is averaged across the prefix-test phases:
   - `≥ 0.5` → `yes`
   - `≥ 0.1` → `partial`
   - otherwise → `no`
2. **Fallback signal** — if the engine does not report `cached_tokens`
   (LM Studio, vllm-mlx, oMLX), TTFT ratio is used: prefix-test TTFT
   versus cold TTFT.
   - `< cold/5` → `yes`
   - `< cold/2` → `partial`
   - otherwise → `no`

### Quality gates

Three gates run alongside the bench and surface in `result["quality_gates"]`:

- **`early_stop`** — flags phases where `completion_tokens` drops below
  50% of the requested `max_tokens` on two or more runs. Catches engine
  bugs where a speculatively-drafted EOS token is accepted incorrectly
  under prefix cache reuse — the result still parses as valid
  OpenAI-compat but the engine silently returns truncated answers.
- **`memory_pressure`** — a background thread polls `vm_stat` and
  `vm.swapusage` every 15s with the baseline taken at bench start. Alerts
  when swap usage grows >500 MB or swapouts grow >1000 from baseline.
  Both indicate the OS is paging the model or KV cache to disk, so the
  measured `tok/s` no longer represents the engine itself.
- **`duplicate_processes`** — a single `ps` snapshot before the bench
  rejects runs where two instances of the same engine are bound, since
  one will compete with the bench for GPU and confuse process attribution.

When a gate trips, the CLI prints a red warning under the verdict line
and the JSON output keeps full per-sample detail so a leaderboard or
regression tracker can refuse to publish the result.

### Reproducible cold starts (opt-in `aisctl` integration)

`asiai bench --agentic-mode --agentic-auto-restart` calls
`aisctl restart <engine>` before the first phase and polls `/health`
until ready. Useful for engines without a model-unload API (llama.cpp,
oMLX, TurboQuant) where a daemon restart is the only reliable way to
wipe the KV cache. Add `--agentic-auto-restart-required` to abort
instead of proceeding when `aisctl` is unavailable.

This integration requires
[`asiai-inference-server`](https://github.com/druide67/asiai-inference-server)
installed; otherwise the bench logs a warning and proceeds against
whatever the engine state already is.

### Why it matters

A single-shot `tok/s` number is meaningless for agent workflows when the
engine does not reuse the system prefix. Two engines with identical
single-shot throughput can differ by **5-10× on agent tick latency**
depending on whether the prefix cache holds.

`agentic-mode` exposes that gap explicitly so the leaderboard and
engine-selection decisions reflect the dominant workload, not a
microbenchmark.

## Environment Safety

asiai performs pre-benchmark checks:

1. **Memory pressure**: refuses to start if critical
2. **Thermal throttling**: warns if speed limit < 80%
3. **Duplicate processes**: warns if multiple instances of the same engine are running (e.g., two `ollama serve` processes on the same port)
4. **Engine runner type**: for Ollama, detects whether `--mlx-engine` or `--ollama-engine` runner is active

These checks prevent measurement errors caused by resource contention or incorrect routing.

## Conformance

| Practice | Status |
|----------|--------|
| Pre-flight gate check (memory pressure + thermal) | Implemented |
| Duplicate process detection | Implemented (v1.5.0) |
| Ollama runner type detection (MLX vs llama.cpp) | Implemented (v1.5.0) |
| TTFT separated from tok/s | Implemented |
| TTFT source labeling (server vs client) | Implemented (v1.5.0) |
| Deterministic sampling (temperature=0) | Implemented |
| Token count from server API (not SSE chunks) | Implemented (warning on fallback) |
| Per-engine power monitoring (IOReport, no sudo) | Implemented |
| 1 warmup generation per engine | Implemented |
| Default 3 runs (SPEC minimum) | Implemented |
| Median as primary metric (SPEC standard) | Implemented |
| Pooled intra-prompt stddev (Bessel N-1) | Implemented (corrected v1.5.0) |
| Model unloading between engines | Implemented |
| Adaptive cooldown (memory pressure-aware) | Implemented |
| Sanity checks (tok/s, TTFT bounds) | Implemented |
| Thermal throttling detection + warning | Implemented |
| Thermal drift detection (monotone decrease) | Implemented |
| Engine version + runner type stored per result | Implemented (v1.5.0) |
| Universal VRAM via ri_phys_footprint | Implemented |
| Historical regression detection | Implemented |
| Dual TTFT measurement (server + client) | Implemented (v1.6.0) |
| Cross-validation script (3 methods compared) | Available (scripts/cross-validate-bench.py) |

## Apple Silicon Considerations

### Unified Memory

Apple Silicon shares memory between CPU and GPU. asiai runs engines **sequentially** and **unloads models between engines** to avoid memory contention and swapping. VRAM is reported natively by Ollama and LM Studio; for other engines, asiai estimates memory usage via `ri_phys_footprint` (the macOS physical footprint metric, same as Activity Monitor). Estimated values are labeled "(est.)" in the UI.

### Thermal Throttling

- **MacBook Air** (no fan): severe throttling under sustained load
- **MacBook Pro** (fan): mild throttling
- **Mac Mini/Studio/Pro**: active cooling, minimal throttling

asiai records `thermal_speed_limit` per result and warns if throttling is detected.

### KV Cache

Large context sizes (32k+) can cause instability on engines that pre-allocate KV cache. Set engine context length to match the actual test size for fair results.

## Power Measurement

asiai measures GPU, CPU, ANE and DRAM power consumption via Apple's IOReport Energy Model framework — **no sudo required**. Power is measured automatically in every benchmark and every monitoring snapshot.

IOReport reads the same hardware energy counters as `sudo powermetrics`, but through a user-space API (`libIOReport.dylib` via ctypes). This eliminates the need for passwordless sudo configuration.

### Validation

We cross-validated IOReport against `sudo powermetrics` under LLM inference load on M4 Pro 64GB, using 10 paired samples per engine at 2-second intervals:

| Engine | IOReport avg | powermetrics avg | Mean delta | Max delta |
|--------|-------------|-----------------|------------|-----------|
| LM Studio (MLX) | 12.6 W | 12.6 W | 0.9% | 2.1% |
| Ollama (llama.cpp) | 15.6 W | 15.4 W | 1.3% | 4.1% |

Both engines confirmed <1.5% average delta with 10/10 paired samples. ANE power was 0.000W across all 20 samples, confirming no LLM engine currently uses the Neural Engine.

The `--power` flag enables additional cross-validation by running both IOReport and `sudo powermetrics` simultaneously, storing both readings for comparison.

### Power Efficiency

Power efficiency (tok/s per watt) is calculated as `tok_per_sec / gpu_watts` for each benchmark result. This metric enables comparison of inference cost across engines and hardware.

## Metadata

Every benchmark result stores: engine, engine_version, model, model_format, model_quantization, hw_chip, os_version, thermal_level, thermal_speed_limit, power_watts, power_source, metrics_version. This enables fair regression comparison and cross-machine benchmarks.



---

## /metrics-spec

Raw markdown: https://asiai.dev/markdown/metrics-spec.md
Rendered: https://asiai.dev/metrics-spec/

---
description: "Detailed definitions of all asiai benchmark metrics: tok/s, TTFT, power watts, efficiency, VRAM, stability, thermal state."
---

# Benchmark Metrics Specification

> **Version**: 0.4.0
> **Status**: Implemented
> **Scope**: `asiai bench` — all engines

## Motivation

Benchmark results must be **comparable across engines**. Each metric has a single definition
that all engine implementations must respect. The implementation may vary (server-side API vs
client-side measurement), but the semantic must be identical.

## Metrics

### M1. `tok_per_sec` — Generation Speed

**Definition**: Tokens produced per second of **generation time only**, excluding prompt
processing (TTFT).

```
generation_s = total_duration_s - ttft_s
tok_per_sec  = tokens_generated / generation_s    (if generation_s >= 0.01)
             = 0.0                                 (otherwise)
```

| Engine | `generation_s` source |
|--------|----------------------|
| Ollama | `eval_duration / 1e9` (server API — direct) |
| OpenAI-compat | `elapsed_s - (ttft_ms / 1000)` (client-side) |

**Rationale**: At large context sizes (e.g. 64k tokens), TTFT can dominate total duration.
Including it in tok/s makes fast generators appear slow (e.g. 3.2 tok/s instead of 42 tok/s).

### M2. `ttft_ms` — Time to First Token

**Definition**: Time between sending the request and receiving the first output token, in ms.

| Engine | Source |
|--------|--------|
| Ollama | `prompt_eval_duration / 1e6` (server API) |
| OpenAI-compat | `(time.monotonic() at 1st content chunk - t0) * 1000` (client) |

Note: Semantics differ slightly (server vs client measurement), but on localhost the gap is
~1ms — acceptable.

### M3. `total_duration_ms` — Total Duration

**Definition**: Wall-clock total request time (prompt processing + generation), in ms.

**Invariant**: `total_duration_ms >= ttft_ms` — always.

| Engine | Source |
|--------|--------|
| Ollama | `total_duration / 1e6` (server API) |
| OpenAI-compat | `elapsed_s * 1000` (client wall-clock) |

### M4. `tokens_generated` — Token Count

**Definition**: Number of output tokens produced by the model.

**Source (by priority)**:
1. Server counter: Ollama `eval_count`, OpenAI-compat `usage.completion_tokens`
2. Text length estimate: `max(1, len(text) // 4)` (heuristic: ~4 chars/token)
3. **Never** `len(text_parts)` (SSE chunks != tokens)

### M5. `generation_duration_ms` — Generation Duration

**Definition**: Generation time only (excluding TTFT), in ms.
Makes the decomposition `total = ttft + generation` explicit and auditable.

| Engine | Source |
|--------|--------|
| Ollama | `eval_duration / 1e6` (server API — direct) |
| OpenAI-compat | `max(0, elapsed_s - ttft_s) * 1000` (computed) |

### M6. `power_watts` — GPU Power

**Definition**: Average GPU power during execution of **this specific engine**, in watts.

**Scope**: One `PowerMonitor` per engine. Started before the first prompt, stopped after
the last run. Each engine gets its own measurement — no session-wide averaging.

Source: `sudo powermetrics` (macOS).

### M7. `tok_per_sec_per_watt` — Energy Efficiency

```
tok_per_sec_per_watt = tok_per_sec / power_watts
```

Uses the corrected tok/s (M1) and per-engine power (M6).

### M8. `std_dev_tok_s` — Variance (Pooled)

**Definition**: Pooled intra-prompt standard deviation — captures run-to-run noise
**without** mixing in inter-prompt variance.

```
For each prompt_type p with runs [v1, v2, ..., vn]:
    var_p = sum((vi - mean_p)^2) / n    (population variance)

pooled_variance = mean(var_p for all p with n >= 2)
std_dev_tok_s   = sqrt(pooled_variance)
```

**Stability classification** (unchanged):
- CV < 5% → `stable`
- CV < 10% → `variable`
- CV >= 10% → `unstable`

Where CV = `(std_dev_tok_s / avg_tok_s) * 100`.

## Implementation Map

| Metric | `base.py` | `ollama.py` | `openai_compat.py` | `runner.py` | `reporter.py` |
|--------|-----------|-------------|--------------------|-----------  |----------------|
| M1 tok/s | field | server API | client (excl. TTFT) | passthrough | avg |
| M2 ttft_ms | field | server API | client streaming | passthrough | avg |
| M3 total_duration_ms | field | server API | client wall-clock | passthrough | avg |
| M4 tokens_generated | field | server API | server or `len//4` | passthrough | avg |
| M5 generation_duration_ms | field | server API | computed | stored in dict | — |
| M6 power_watts | — | — | — | per-engine monitor | passthrough |
| M7 tok/s/W | — | — | — | computed | passthrough |
| M8 std_dev | — | — | — | — | pooled intra-prompt |

## Benchmark Protocol

1. **Warmup**: 1 non-timed generation per engine (`"Hello"`, max_tokens=1) to prime caches.
2. **Measured runs**: Default 3 runs per prompt per engine (configurable via `--runs`).
3. **Sampling**: `temperature=0` (greedy) on all engines for deterministic output.
4. **Reporting**: Median tok/s as primary metric (SPEC standard), mean +/- stddev as secondary.
5. **Throttling**: Warning emitted if `thermal_speed_limit < 100%` during any run.
6. **Metadata**: engine_version, model_format, model_quantization, hw_chip, os_version
   stored per result for reproducibility.

See [benchmark-best-practices.md](benchmark-best-practices.md) for full methodology audit.



---

## /ollama-vs-lmstudio

Raw markdown: https://asiai.dev/markdown/ollama-vs-lmstudio.md
Rendered: https://asiai.dev/ollama-vs-lmstudio/

---
title: "Ollama vs LM Studio: Apple Silicon Benchmark"
description: "Ollama vs LM Studio benchmark on Apple Silicon: tok/s, TTFT, power, VRAM compared side by side on M4 Pro with real measurements."
type: article
date: 2026-03-28
updated: 2026-03-29
dataset:
  name: "Ollama vs LM Studio Benchmark on Apple Silicon M4 Pro"
  description: "Head-to-head benchmark comparing Ollama (llama.cpp) and LM Studio (MLX) on Mac Mini M4 Pro 64GB with Qwen3-Coder-30B. Metrics: tok/s, TTFT, GPU power, efficiency, VRAM."
  date: "2026-03"
---

# Ollama vs LM Studio: Apple Silicon Benchmark

Which inference engine is faster on your Mac? We benchmarked Ollama (llama.cpp backend) and LM Studio (MLX backend) head-to-head on the same model and hardware using asiai 1.4.0 in March 2026.

## Test Setup

| | |
|---|---|
| **Hardware** | Mac Mini M4 Pro, 64 GB unified memory |
| **Model** | Qwen3-Coder-30B (MoE architecture, Q4_K_M / MLX 4-bit) |
| **asiai version** | 1.4.0 |
| **Methodology** | 1 warmup + 1 measured run per engine, temperature=0, model unloaded between engines ([full methodology](methodology.md)) |

## Results

| Metric | LM Studio (MLX) | Ollama (llama.cpp) | Difference |
|--------|-----------------|-------------------|------------|
| **Throughput** | 102.2 tok/s | 69.8 tok/s | **+46%** |
| **TTFT** | 291 ms | 175 ms | Ollama faster |
| **GPU Power** | 12.4 W | 15.4 W | **-20%** |
| **Efficiency** | 8.2 tok/s/W | 4.5 tok/s/W | **+82%** |
| **Process Memory** | 21.4 GB (RSS) | 41.6 GB (RSS) | -49% |

!!! note "About memory numbers"
    Ollama pre-allocates KV cache for the full context window (262K tokens), which inflates its memory footprint. LM Studio allocates KV cache on demand. The process RSS reflects total memory used by the engine process, not just model weights.

## Key Findings

### LM Studio wins on throughput (+46%)

MLX's native Metal optimization extracts more bandwidth from Apple Silicon's unified memory. On MoE architectures, the advantage is significant. On the larger Qwen3.5-35B-A3B variant, we measured an even wider gap: **71.2 vs 30.3 tok/s (2.3x)**.

### Ollama wins on TTFT

Ollama's llama.cpp backend processes the initial prompt faster (175ms vs 291ms). For interactive use with short prompts, this makes Ollama feel snappier. For longer generation tasks, LM Studio's throughput advantage dominates total time.

### LM Studio is more power-efficient (+82%)

At 8.2 tok/s per watt vs 4.5, LM Studio generates nearly twice as many tokens per joule. This matters for laptops on battery and for sustained workloads on always-on servers.

### Memory usage: context matters

The large gap in process memory (21.4 vs 41.6 GB) is partly due to Ollama pre-allocating KV cache for its maximum context window. For a fair comparison, consider the actual context used during your workload, not the peak RSS.

## When to Use Each

| Use Case | Recommended | Why |
|----------|------------|-----|
| **Maximum throughput** | LM Studio (MLX) | +46% faster generation |
| **Interactive chat (low latency)** | Ollama | Lower TTFT (175 vs 291 ms) |
| **Battery life / efficiency** | LM Studio | 82% more tok/s per watt |
| **Docker / API compatibility** | Ollama | Broader ecosystem, OpenAI-compat API |
| **Memory-constrained (16GB Mac)** | LM Studio | Lower RSS, on-demand KV cache |
| **Multi-model serving** | Ollama | Built-in model management, keep_alive |

## Other Models

The throughput gap varies by model architecture:

| Model | LM Studio (MLX) | Ollama (llama.cpp) | Gap |
|-------|-----------------|-------------------|-----|
| Qwen3-Coder-30B (MoE) | 102.2 tok/s | 69.8 tok/s | +46% |
| Qwen3.5-35B-A3B (MoE) | 71.2 tok/s | 30.3 tok/s | +135% |

MoE models show the largest differences because MLX handles sparse expert routing more efficiently on Metal.

## Run Your Own Benchmark

```bash
pip install asiai
asiai bench --engines ollama,lmstudio --prompts code --runs 3 --card
```

asiai compares engines side by side with the same model, same prompts, and same hardware. Models are automatically unloaded between engines to prevent memory contention.

[View the full methodology](methodology.md) · [See the community leaderboard](leaderboard.md) · [How to benchmark LLMs on Mac](benchmark-llm-mac.md)



---

## /qwen38-27b-apple-silicon

Raw markdown: https://asiai.dev/markdown/qwen38-27b-apple-silicon.md
Rendered: https://asiai.dev/qwen38-27b-apple-silicon/

---
title: "Qwen3.8-27B on Apple Silicon: +45% for One Flag"
description: "Qwen3.8-27B ships its speculative decoding head inside the GGUF, disabled by default. Measured on M5 Max: 21.9 to 31.8 tok/s with one flag, and why the draft cap inherited from Qwen3.6 gains you nothing."
type: article
date: 2026-08-15
updated: 2026-08-15
---

# Qwen3.8-27B on Apple Silicon: +45% for One Flag

Qwen3.8-27B ships its own **multi-token prediction head inside the GGUF file**. No
separate draft model to download, no extra memory. llama.cpp only loads it when
you pass `--spec-type draft-mtp` — without the flag it drops the tensors as
"unused" and says nothing.

Measured on an M5 Max, same model, same quantization, everything else identical:

```bash
--spec-type draft-mtp --spec-draft-n-max 4
```

**21.9 → 31.8 tok/s.** That is a 45% gain for one line of configuration.

## The Draft Cap Is Where People Will Lose It

The flag alone is not enough, and this is the part that will cost most people
their gain. `--spec-draft-n-max` sets how many tokens the draft head proposes per
step, and it has an **optimum**:

| Draft cap | Throughput (warm) | |
|---|---|---|
| speculation off | 21.9 tok/s | baseline |
| `--spec-draft-n-max 2` | 22.2 tok/s | **the Qwen3.6 value — gains nothing** |
| **`--spec-draft-n-max 4`** | **31.8 tok/s** | **+45%** |
| `--spec-draft-n-max 6` | 21.6 tok/s | back to square one |

**2 is the value carried over from Qwen3.6 presets.** Anyone migrating will turn
speculation on, measure a 1% gain, and conclude that MTP is not worth it on this
model. It is worth 45% — at cap 4.

Why 6 loses: the cost of producing the draft grows linearly with the cap, while
acceptance saturates. Measured at cap 4 the model accepts 54.8% of drafted tokens;
at cap 6, 48.5% — for 23% more tokens drafted. The extra work is not paid back.

A note for anyone tempted to tune further: **the acceptance rate does not tell you
whether the cap is right**. At cap 2 acceptance was an excellent 75% while the
engine was leaving 45% of its throughput on the table. Measure throughput, not
acceptance.

## Context and KV Cache: 23% at Depth

Two settings interact, and the winning combination beats each of its parts.

Quantizing the KV cache saves memory and costs throughput at depth:

| KV cache | Throughput at 56k ctx | Resident memory |
|---|---|---|
| `q8_0` | 18.5 tok/s | 36.0 GB |
| **`f16`** | **20.6 tok/s** | 44.8 GB |

Halving the context window from 262144 to 131072 costs **no throughput at all**
and saves 4.8 GB. Combine the two:

| Configuration | Warm | At 56k ctx | Memory |
|---|---|---|---|
| ctx 262144 · KV `q8_0` | 29.6 | 18.5 | 36.0 GB |
| **ctx 131072 · KV `f16`** | **31.8** | **22.7** | **36.8 GB** |

**+23% throughput at depth for 800 MB.** An `f16` cache over half the context
weighs what a quantized cache weighs over the full one. If your workload never
exceeds 128k tokens, this is free performance.

## What This Costs You

Dropping to 131072 halves the model's **native** context (262144, extensible to
1M). If you actually work beyond 128k tokens, keep the full window — the
measurement says the KV cache precision matters, the window size does not.

## Reproducing This

```bash
# with speculation
llama-server --model Qwen3.8-27B-UD-Q5_K_XL.gguf \
  --ctx-size 131072 --flash-attn on --n-gpu-layers 999 --jinja \
  --cache-type-k f16 --cache-type-v f16 \
  --spec-type draft-mtp --spec-draft-n-max 4 --metrics

# then verify it actually loaded — the witness is in the server log
grep "creating MTP draft context" <server.log>

# and confirm it is working, from the artifact rather than the log
curl -s localhost:8080/metrics | grep spec_decode
```

That last check matters: `llamacpp:spec_decode_num_draft_tokens_total` divided by
`llamacpp:spec_decode_num_drafts_total` gives the tokens drafted per step. It
should equal your cap. If it does not, the flag did not take.

## Conditions

M5 Max 128 GB, macOS 26.5.2, mains power, High Power Mode, llama.cpp b10434,
`Qwen3.8-27B-UD-Q5_K_XL`, agentic protocol n=3, reasoning disabled and verified,
single resident engine, production stopped. Thermal throttling to 50% was present
on every measurement (expected M5 behaviour after 80-110 s of dense generation)
and is therefore neutral for comparisons within this table — but it makes these
numbers non-comparable to figures published elsewhere without the same conditions.

Depth matters: the same engine drops from 31.8 tok/s at 7.5k prompt tokens to
22.7 at 56k. A tok/s figure without its context depth means nothing.

## See Also

- [Choosing an Engine for Qwen3.8-27B](qwen38-27b-engine-choice.md) — four engines
  measured, and why the fastest one is unusable for agents
- [Agentic Benchmarks](agentic-benchmarks.md) — the protocol behind these numbers
- [Benchmark Best Practices](benchmark-best-practices.md)



---

## /qwen38-27b-engine-choice

Raw markdown: https://asiai.dev/markdown/qwen38-27b-engine-choice.md
Rendered: https://asiai.dev/qwen38-27b-engine-choice/

---
title: "The Best Engine for a Local Agent on Qwen3.8-27B: MTPLX, and it is not close"
description: "Eleven configurations, eight engines, one M5 Max. For agent work the winner takes every metric that matters: 99 ms to first token, token-level prefix reuse, 44 tok/s, 27 GB. And the flag that buys 20-50% is off by default in four of the six engines that support it."
type: article
date: 2026-08-16
updated: 2026-08-16
---

# The Best Engine for a Local Agent on Qwen3.8-27B

**Run MTPLX with `Qwen3.8-27B-MTPLX-Optimized-Speed` and `--mtp --depth 3`.**

We measured eleven configurations across eight engines on one M5 Max. For agent work it
wins on every metric that matters, and the second place is not close:

| | MTPLX Optimized-Speed | llama.cpp b10434 +MTP | Ollama 0.32.13 |
|---|---|---|---|
| First token | **99 ms** | 125 ms | 240 ms |
| First token, 56k cold | **301 ms** | 365 ms | 574 ms |
| Prefix reuse | **7,530 / 7,530** | 7,526 / 7,530 | 0 / 7,530 |
| Throughput | **44.4 tok/s** | 31.8 | 32.8 |
| Memory | **27 GB** | 36.8 GB | 30.3 GB |

That is +40% throughput, 26 ms less latency per turn, and 10 GB less memory than the
best of the rest. We run it in production.

## Why those metrics and not tokens per second

An agent is not a chat. It does not stream one long answer — it takes dozens of short
turns, and **every turn re-reads everything that came before**. So the number you feel is
time to first token, paid once per turn, and it depends almost entirely on whether the
engine kept the previous prompt in cache.

The spread on that metric is **335×** across our table — 99 ms to 33 seconds. The spread
on throughput is 5%. That is the whole argument: for agent work, pick on latency and
prefix reuse; throughput is a tiebreaker.

Two engines reuse at token level (MTPLX, llama.cpp). One reuses in 1,024-token blocks and
re-prefills 1,386 tokens *every single turn, forever* (oMLX, 1,968 ms). Three reuse
nothing at all and pay the full prompt each time — mlx-vlm at 9,982 ms, rapid-mlx at
33,195 ms. On a 60-turn agent session, that last one costs **33 minutes of pure waiting**.

## The full table

Warm and 56k are tok/s. First token on the warm turn; @56k on a cold 55,839-token prompt.

| Engine | Weights | Warm | At 56k | First token | @56k | Memory | tok/s/W |
|---|---|---|---|---|---|---|---|
| MTPLX Bare-Speed | MTPLX 4-bit g64 | **56.0** | **46.5** | 101 ms | 322 ms | 22.0 GB | 0.694 |
| **MTPLX Optimized-Speed** | MTPLX 4-bit g32 | 44.4 | 37.5 | **99 ms** | **301 ms** | 27.0 GB | 0.623 |
| mlx-vlm + MTP drafter | MLX 4-bit ⁽¹⁾ | 43.3 | 28.9 | 9,982 ms | 100,733 ms | 15.5 GB | **0.744** |
| MTPLX Optimized-Quality | MTPLX 8-bit g64 | 40.8 | 30.3 | 118 ms | 100,243 ms | 32.9 GB | 0.575 |
| Ollama 0.32.13 | GGUF (undeclared) | 32.8 | 21.7 | 240 ms | 574 ms | 30.3 GB | 0.451 |
| llama.cpp b10434 + MTP | GGUF Q5_K_XL ⁽²⁾ | 31.8 | 22.7 | 125 ms | 365 ms | 36.8 GB | 0.395 |
| rapid-mlx 0.12.11 | MLX 4-bit ⁽¹⁾ | 30.2 | 24.7 | 33,195 ms | 287,548 ms | 15.0 GB | 0.544 |
| oMLX 0.6.0-dev | oQ4e-mtp (third-party) | 29.8 | 24.6 | 1,968 ms | 1,991 ms | 16.7 GB | 0.512 |
| mlx-lm 0.31.3 | MLX 4-bit ⁽¹⁾ | 28.8 | 24.0 | 432 ms | 799 ms | **14.6 GB** | 0.472 |
| LM Studio 0.4.21 **+MTP** | GGUF Q5_K_XL ⁽²⁾ | 27.8 | 23.4 | 419 ms | 825 ms | 36.3 GB | 0.422 |
| LM Studio 0.4.21 defaults | GGUF Q5_K_XL ⁽²⁾ | 23.1 | 18.7 | 359 ms | 865 ms | 35.2 GB | 0.351 |

⁽¹⁾ ⁽²⁾ = byte-identical weights file. MTPLX rows ran on 2.6.0. One cell, one export.
All columns are medians of the warm phase. `tok/s/W` divides decode throughput by
**whole-SoC** power (58 to 83 W depending on the engine) — not GPU-only, which ranks
differently.

**How to read it.** Throughput gaps under 8% are noise — Ollama through LM Studio+MTP
(32.8 to 27.8) are tied, not ranked. The two 100-second entries are cache misses, not
engine speed: Optimized-Quality does 118 ms warm and returns to 498 ms on the repeated
56k turn. Memory is not comparable across families (llama.cpp mmaps: 36.8 GB resident is
19.4 GB physical).

## The one thing that will surprise you: the server barely matters

mlx-vlm, mlx-lm and rapid-mlx served the *same file*, `lmstudio-community/Qwen3.8-27B-MLX-4bit`,
snapshot `6067b15c`, byte for byte. Two of them ran bare, without speculation:

**30.2 against 28.8 tok/s. A 4.9% difference — below our own rerun noise.**

Swapping the MLX server buys nothing. The third one, mlx-vlm, reached 43.3 — but it ran
with the MTP drafter loaded. **That +50% is the flag, not the server** — see the appendix
on multi-token prediction, which reaches the same conclusion from the other direction.

## Choosing against a different constraint

Our recommendation optimises for agent work. If yours differs, the table already answers:

- **Smallest footprint** → mlx-lm, 14.6 GB. You lose the prefix cache granularity.
- **Best battery life** → mlx-vlm, 0.744 tok/s/W. Unusable for agents (9,982 ms first
  token), excellent for batch.
- **Raw throughput** → MTPLX Bare-Speed, 56.0 tok/s. We advise against it: the quantization
  author publishes a divergence-from-bf16 of 0.0376 against 0.0220 for Optimized-Speed —
  36× further from bf16 than the 8-bit build. His measurement, not ours, not reproduced.
- **You already run llama.cpp** → stay. Add `--spec-type draft-mtp` and you close most of
  the gap; 125 ms first token is fine for an agent.

## Run it with reasoning on

Qwen recommends `xhigh` for agentic work and warns that a lower effort *"can lead to
insufficient analysis, more failures, and repeated retries"*. MTPLX 2.7.1 separately lists
disabled reasoning as a known issue for Qwen3.8. `high` does not exist on this model —
`low`, `medium`, `xhigh` only, and `xhigh` is the default.

⚠️ Note the effort level is a **chat-template variable, not an API field**: sent as a
normal request parameter it is silently ignored. It must be set at launch.

## Four limits that change how you read this

1. **Nothing here ran a task to completion.** Output was capped at 400 tokens and every run
   hit the cap. These are streaming throughput numbers on truncated continuations.
2. **Reasoning was off for all measurements**, to make engines comparable. That is not how
   we deploy — see above. We have no figure on what reasoning costs each engine.
3. **Tuning was not symmetric.** llama.cpp got explicit cache flags the MLX engines did
   not; MTPLX ran `--profile turbo`; sampling parameters and thermal limits differ per
   engine. Gaps *between families* are indicative, not clean.
4. **We are recommending the engine that alone reads its own quantization format, measured
   with our own tool, published on our own site.** The MTPLX rows are the only ones where
   engine and weight format cannot be separated. Weigh it accordingly — the flag table in
   the appendix is the part of this page that costs us nothing to be right about.

Two engines with native MTP, vmlx and vllm-mlx, could not be measured in time.

## Appendix: the MTP flag, engine by engine

Qwen3.8 ships a multi-token prediction head **inside the weights**. The model proposes
several tokens ahead, the engine verifies them in one pass. Four of the six engines that
support it ship it **off**.

| Engine | Flag | On by default? |
|---|---|---|
| MTPLX ⁽ᵐ⁾ | `--mtp --depth 3` | **yes** |
| vmlx ⁽ᵈ⁾ | `--native-mtp-depth` | **yes** |
| llama.cpp ⁽ᵐ⁾ | `--spec-type draft-mtp --spec-draft-n-max 4` | no |
| LM Studio ⁽ᵐ⁾ | `--speculative-draft-mtp` | no |
| mlx-vlm ⁽ᵐ⁾ | `--draft-kind mtp --draft-model mlx-community/Qwen3.8-27B-MTP-4bit` | no |
| vllm-mlx ⁽ᵈ⁾ | `--enable-mtp` | no |
| Ollama · mlx-lm · rapid-mlx | no support | — |

⁽ᵐ⁾ attested by our own launch commands and logs. ⁽ᵈ⁾ from project documentation only.

**We measured what it costs to not know**: same weights file, same 65,536 context, same
thermal sequence, everything identical but two flags. **LM Studio goes 23.1 → 27.8 tok/s,
+19.9%.** No GUI surfaces this. On mlx-vlm the same head is worth +50%.

Note the drafter is bundled in the GGUF for llama.cpp and LM Studio, but mlx-vlm needs a
**second repository** downloaded alongside.

**Check it is actually engaged** — a flag accepted is not a flag working:

```
# llama.cpp / LM Studio — the log must mention a draft context at startup
grep -i "draft" server.log        # "creating MTP draft context"
# any OpenAI-compatible engine — acceptance rate should sit at 0.9+
curl -s localhost:8080/v1/chat/completions -d '…' | jq '.timings'
```

## Reproducing this

Every number comes from a certified card produced by [asiai](https://asiai.dev), through a
single scripted path with solitude gates, served-model identity proofs and thermal
sampling. Raw exports, full launch commands and prompt text: ask and we publish them.

If you take one thing: **check whether your engine has multi-token prediction, and whether
it is on.**



---

## /research/comparison-panel/2026-05-apple-silicon-agentic-inference-panel

Raw markdown: https://asiai.dev/markdown/research/comparison-panel/2026-05-apple-silicon-agentic-inference-panel.md
Rendered: https://asiai.dev/research/comparison-panel/2026-05-apple-silicon-agentic-inference-panel/

# Apple Silicon Agentic Inference Panel

> Comparative benchmark panel across inference engines (llama.cpp, mlx-lm,
> LM Studio, Rapid-MLX, vLLM-MLX, oMLX, vMLX, Ollama) running Qwen 3.6
> family models on Apple Silicon M-series, measured with
> `asiai bench --agentic-mode` and `asiai bench --burst-mode`.
>
> **Workload target**: agent-orchestrator class — ~60-80 tool calls per turn,
> identical system prompt of ~7 KB, user message changing per call. This is
> the worst case for naïve prefix caching: a true cache-reuse cross-USER is
> required, not just cache-on-the-same-prompt.
>
> **Reading the throughput figures**: Section 1 decode rates use the Qwen3
> default chat template (thinking ON), so they include reasoning tokens —
> effective agent-throughput on a thinking model is lower. Thinking is a
> per-task trade-off (caveat 1), not a global on/off.
>
> Published 2026-06 · contributions and corrections welcome via
> [github.com/druide67/asiai](https://github.com/druide67/asiai/issues).

## ⚠️ Known caveats before reading further

1. **Thinking mode is a per-task trade-off.** With the Qwen3 default template
   (thinking ON), Qwen 3.6 / Qwopus emit ~6-7× more tokens, so the Section 1
   decode figures **include reasoning tokens** and effective agent-throughput is
   lower. Thinking ON is **required** for written multi-section deliverables (a
   thinking-OFF model skips the deliverable) but **costs** atomic tool-call
   cleanliness (asiai measures ~100% clean tool calls with thinking OFF vs
   ~77.8% with thinking ON + `preserve_thinking` ON, deterministic across runs;
   `enable_thinking=on` + `preserve_thinking=off` is unusable — a deterministic
   HTTP 500 once reasoning accumulates in the context). Set thinking **per
   task-dimension**, not as one global flag.
2. **Rapid-MLX and vLLM-MLX share an engine.** Rapid-MLX is a community fork of
   `waybarrios/vllm-mlx`; they appear as separate rows below because they have
   diverged in version and features, but the prefix-cache snapshot mechanism is
   the same lineage.
3. **MTP: Qwen 3.6 has a real head; the backend matters.** Qwen 3.6's official
   `config.json` carries `mtp_num_hidden_layers=1` (Qwen naming — **not** the
   DeepSeek `num_nextn_predict_layers` key, so a `nextn`-only check wrongly
   concludes "no head"). Some re-quantized GGUF/MLX artifacts drop the MTP
   tensors while keeping the config flag — verify the tensors in the weight
   index, not just the flag. llama.cpp native MTP (`--spec-type draft-mtp`)
   **requires a `-MTP-GGUF`** that embeds the head; a plain GGUF cannot draft.
   Released mlx-lm does not run the head as native speculative decoding (PR
   [ml-explore/mlx-lm#990](https://github.com/ml-explore/mlx-lm/pull/990) adds
   it). LM Studio routes GGUF through its llama.cpp-derived backend and MLX
   through `mlx-engine`.
4. **Single-pass measurements, no variance reporting** — Section 1 / 2
   chiffres are single observations. Variance reporting (median + min + max
   across N passes) is supported as of `--burst-runs N` but the rebench
   is pending.

| Section | Topic | Status |
|---------|-------|--------|
| 1 | Single-call performance | 🟡 8 cells, thinking-mode ON (decode includes reasoning tokens) |
| 2 | Concurrent burst (30/60/80 parallel calls) | 🟡 smoke cell + 2 partial concurrent points; no normalized 30/60/80 panel |
| 3 | Caches & optimizations | ✅ 8 engines covered |
| 4 | Memory & resources | ✅ idle + under-load swap (+0) + footprint measured |
| 5 | Model quality (public leaderboards) | 🟡 vendor/self-reported figures (llm-stats) |
| — | **asiai direct measurements** | ✅ dev-quality, thinking ablation, MTP, instruction-following |
| 6 | Operational (license, endpoints, maintenance) | ✅ 8 engines covered |
| 7 | Quality benchmark weighting | 🟡 default weighting, override via `--weights` planned |
| 8 | Custom long-horizon eval (proposal) | 🟡 scoped, not yet built |

---

## Section 1 — Single-call performance

> 🟠 **May 2026 snapshot — indicative, not the reference numbers.** This table was
> captured in May (thinking-mode ON, single-pass) and its source fixtures have not
> been re-verified. For **current, reproducible decode throughput**, use the *asiai
> direct measurements* section below (June, llama.cpp b9430, deterministic). What
> this table is reliable for is the **relative TTFT / prefix-cache** story
> (cross-USER reuse), not absolute t/s. Note in particular that the 123.9 t/s in
> row 5 (LM Studio GGUF+MTP) sits right next to the June **llama.cpp Qwopus+MTP
> 123.3 t/s** — LM Studio's GGUF path is a llama.cpp-derived backend, so the two
> measure essentially the same engine.

> ⚠️ **Read with caveat 1 above**: every figure in this table includes the
> Qwen3 default thinking-mode tokens (reasoning_content). Effective
> agent-throughput requires re-running with
> `chat_template_kwargs={"enable_thinking": false}`. The column is labeled
> "decode (t/s)" not "effective throughput".
>
> The "lower-bound estimate" column is `60 × (TTFT + max_tokens/decode)`,
> assuming sequential dispatch (which Rapid-MLX single-slot enforces). It is
> **not** a production tick prediction — see [Section 7](#section-7) for the
> methodological caveat.
>
> 📌 **Versions tested (May 2026)**: Rapid-MLX 0.6.66, LM Studio 0.4.14,
> llama.cpp b9270. Engine versions churn weekly on Apple Silicon — treat each
> figure as dated, not current. (The asiai-measurements section uses llama.cpp
> b9430.)

| # | Engine | Model | Format | Warm decode (t/s) ¹ | TTFT warm (ms) | TTFT prefix-test median (ms) | TTFT cold (ms) | Lower-bound estimate (60 calls × single-call, optimistic) | Source fixture |
|---|--------|-------|--------|--------------------:|---------------:|----------------------------:|---------------:|----------------------------------------------------------:|----------------|
| 1 | Rapid-MLX 0.6.66 (fork of vllm-mlx) | Qwopus 3.6-35B-A3B-v1 (zaydiscold MLX-4bit) | MLX-4bit | **109.1** ¹ | 139 | **131** | 2074 | ~3.6 min | `cell-rapidmlx-qwopus35b.json` |
| 2 | Rapid-MLX 0.6.66 | Qwen 3.6-35B-A3B-UD (MLX-4bit) | MLX-4bit | 106.9 ¹ | 321 | 319 | 2095 | ~4 min | `cell-rapidmlx-35b-a3b.json` |
| 3 | Rapid-MLX 0.6.66 | Qwopus 3.6-27B-v2 (Jackrong MLX-4bit) | MLX-4bit | 31.8 ¹ | 323 | 323 | 8647 | ~13 min | `cell-rapidmlx-qwopus.json` |
| 4 | Rapid-MLX 0.6.66 | Qwen 3.6-27B-UD (MLX-4bit) | MLX-4bit | 20.5 ¹ | 527 | 527 | 8954 | ~23 min | `cell-rapidmlx-full-27bud.json` |
| 5 | LM Studio 0.4.14 (GGUF backend) ² | Qwen 3.6-35B-A3B-MTP (Unsloth GGUF) | GGUF Q4 + MTP | **123.9** ¹ ² | 309 | 5965 | 6063 | ~3.5 min warm / ~9.2 min prefix-changing | `cell-lmstudio-mtp-qwen35b.json` |
| 6 | LM Studio 0.4.14 (GGUF backend) ² | Qwopus 3.6-35B-A3B-v1 (Jackrong GGUF) | GGUF Q4_K_S | 105.6 ¹ | 292 | 5785 | 5624 | ~3.5 min warm / ~9.6 min prefix-changing | `cell-lmstudio-qwopus35b.json` |
| 7 | llama.cpp b9270 | Qwen 3.6-35B-A3B (UD Q5_K_XL) | GGUF Q5_K_XL | 80.9 ¹ | 3000 | 3000 | n/a | ~8 min | (baseline reference) |
| 8 | llama.cpp b9270 | Qwopus 3.6-27B-v2 (Jackrong GGUF Q4) | GGUF Q4 | 25.3 ¹ | 13000 | 13000 | n/a | ~30 min | (baseline reference) |

¹ **Thinking-mode caveat**: figures captured with default chat template
(thinking ON). Real-world effective throughput on tool-call workloads is
typically 4-12 t/s on Qwopus/Qwen3.6 finetunes when reasoning tokens
inflate output 6-7×. To reproduce these decode figures, pass
`chat_template_kwargs={"enable_thinking": false}` in the request payload.

² **LM Studio backend**: rows 5-6 used a GGUF file, which routes through
LM Studio's llama.cpp-derived backend (NOT the MLX runtime `mlx-engine`).
The MTP claim in row 5 reflects this backend's implementation, not
mlx-engine speculative decoding. Released mlx-lm does not run the MTP head
as native speculative decoding (its `sanitize()` historically dropped MTP
weights during conversion; native support is in PR
[ml-explore/mlx-lm#990](https://github.com/ml-explore/mlx-lm/pull/990)),
so a hypothetical MLX-format MTP model would not benefit on the released
mlx-engine either.

### Key observations

- On the realistic agent pattern (identical system + changing user prompts),
  **Rapid-MLX + Qwopus 35B-A3B-v1** delivers 131 ms median TTFT prefix-test
  vs 5965 ms for LM Studio GGUF backend (**~44× faster**). The advantage
  comes from the vllm-mlx prefix-cache snapshot mechanism (see Section 3
  for the source-code disambiguation).
- On pure decode throughput (warm path), the **LM Studio GGUF backend with
  Unsloth MTP** records 123.9 t/s vs Rapid-MLX 109.1 t/s (+13.5%). This delta
  reflects the LM Studio llama.cpp-derived backend's speculative decoding on a
  GGUF carrying the MTP head, not an Apple-MLX gain (released mlx-engine does
  not run the head — see footnote 2). On the native llama.cpp path, MTP is
  net-positive on the MoE 35B-A3B — see Section 3.
- All `Qwen 3.6 family` configurations (hybrid DeltaNet + full-attention) fail
  cross-USER prefix cache **except Rapid-MLX**, which keeps an RNN-state
  snapshot. On llama.cpp / LM Studio GGUF `llama_memory_can_shift=false`; on
  mlx-lm / oMLX the recurrent/SSM state can't be split at an arbitrary token
  boundary. The upstream llama.cpp fix for this architecture is not merged
  ([#23121](https://github.com/ggml-org/llama.cpp/pull/23121) closed;
  `preserve_thinking` does not address it,
  [#22615](https://github.com/ggml-org/llama.cpp/issues/22615)).
- **Single-slot serialization confirmed**: smoke burst test (Section 2)
  shows Rapid-MLX 0.6.66 serializes concurrent calls FIFO (p50 ≈ p95 ≈ max
  on burst=5). For 60-80 calls/turn, total wall-time scales linearly with
  burst size on this engine. A multi-slot engine (e.g. llama.cpp
  `--parallel N`) would behave differently, but `--parallel N` on Qwen3.6
  hybrid disables prefix cache per slot (architectural limitation).

---

## Section 2 — Concurrent burst (30/60/80 parallel calls)

> Pattern: 30 to 80 concurrent `POST /v1/chat/completions` calls fired within a
> ~200 ms window. Simulates an agent loop dispatching multiple MCP/tool calls in
> parallel. Measured natively via `asiai bench --burst-mode`.
>
> 🟡 **Status**: 1 smoke cell measured (Rapid-MLX burst-5). Full panel pending.

### Smoke cell (Rapid-MLX 0.6.66 + Qwopus 35B-A3B-v1, burst=5)

| burst N | wall-time (s) | p50 latency (ms) | p95 latency (ms) | max latency (ms) | agg throughput (t/s) |
|--------:|--------------:|-----------------:|-----------------:|-----------------:|---------------------:|
| 5 | 2.8 | 2615 | 2792 | 2812 | 88.8 |

**Smoke finding**: `p50 ≈ p95 ≈ max` indicates the 5 calls were **serialized
server-side** (single-slot engine). Rapid-MLX 0.6.66 does **not** appear to support
concurrent request scheduling — calls queue FIFO internally. To validate at 60/80
calls scale.

### Full concurrent panel — not yet measured

A normalized 30/60/80-concurrent panel has not been run (the measurements here are
sequential agentic-mode, not concurrent burst). The two partial concurrent data
points that exist elsewhere:

- **TurboQuant** (K=`q8_0` V=`turbo2`, Qwen3-4B, M4 Pro): **+9% aggregate at
  4-parallel** (68.5 → 74.7 t/s) even though single-stream is −8% — the KV
  compression buys back the parallel headroom.
- **oMLX** continuous batching (mlx-lm `BatchGenerator`): **×1.8 aggregate at
  burst-8** (12.8 → 22.9 t/s), but it **collapses at burst-30** (17.3 t/s) once a
  27B-dense saturates RAM into swap — 0 crashes.

A dedicated burst-mode panel across all engines is deferred.

---

## Section 3 — Caches & optimizations

| # | Couple | Cache reuse cross-USER | Snapshot persists cross-restart | MTP support | MTP accept rate | TurboQuant compat | KV cache native types | Native parallel slots |
|---|--------|---|---|---|---|---|---|---|
| 1 | Rapid-MLX + Qwopus 35B-A3B-v1 | ✅ YES (RNN-state snapshot, see ³ below) | ✅ persistent in `~/.cache/vllm-mlx/` | ❌ released MLX runtime doesn't run the MTP head as speculative decode (mlx-lm PR #990 pending) | n/a | ❌ MLX only | MLX native (no quant flag exposed) | ⚠️ single slot (smoke burst confirms FIFO serialization) |
| 2 | Rapid-MLX + Qwen 35B-A3B-UD | ✅ YES ³ | ✅ persistent | ❌ | n/a | ❌ | MLX native | ⚠️ single slot |
| 3 | LM Studio + Qwen 35B-A3B-MTP | ❌ NO (architectural hybrid limitation) | n/a | ✅ via mlx-engine v1.8.1 | **82.1 %** (on coding task) | ❌ | mlx-engine v1.8.1 (4bit MLX) | configurable via GUI |
| 4 | LM Studio + Qwopus 35B-A3B-v1 | ❌ NO | n/a | ❌ no heads | n/a | ❌ | mlx-engine v1.8.1 (Q4_K_S GGUF) | configurable via GUI |
| 5 | llama.cpp + Qwen 3.6-35B-A3B | ❌ NO (architectural hybrid limitation) | n/a | ✅ `--spec-type draft-mtp` on a `-MTP-GGUF` (a plain GGUF cannot draft). Net-positive on the MoE 35B-A3B — asiai measures **+38%** decode (base) / **+17%** (Qwopus) on M5 Max (see § asiai measurements) | benefit = intra-session decode delta (no acceptance rate logged) | ✅ turbo2/3/4 V cache | `fp16`, `q8_0`, `q5_0`, `turbo2/3/4` | ⚠️ `--parallel N` works mechanically but **disables prefix cache per slot on hybrid arch** (each slot owns its KV, the `--cache-reuse N` flag is already silently disabled here). Use with caution. |
| 6 | mlx-lm | ❌ NO (PRs #923, #188, #192 pending upstream) | n/a | ❌ broken on hybrid arch | n/a | ❌ | MLX native | ❌ (single slot) |
| 7 | oMLX | ❌ NO (tool calling lost post-cache-hit, issue #825) | partial | ❌ | n/a | ❌ | MLX native + tiered SSD cache | ❌ |
| 8 | vLLM-MLX (`waybarrios`, upstream of Rapid-MLX) | ⚠️ trie prefix-cache, no documented hybrid/DeltaNet support (Rapid-MLX rows 1-2 add the RNN-state snapshot on top) | n/a | ⚠️ MTP added in prerelease 0.4.0rc1 | n/a | ❌ | MLX + paged-attention | ✅ |

³ **Rapid-MLX prefix cache**: the cache stores hybrid-attention KV slabs +
RNN-state snapshots, keyed per `<repo>--<sys_prompt_hash>` and persisted under
`~/.cache/vllm-mlx/`. The observed ~131 ms TTFT prefix-test is an in-RAM KV slab
reattach plus the changed-user forward pass, not a from-disk reload.

**oMLX large-context cache.** oMLX's 2-tier paged SSD KV cache turns a 55K-token
prefill from ~115 s to ~**3.5 s** TTFT on a same-prompt cache-hit (×33; 55,296 /
55,837 tokens cached). On small prompts (~7.5K) there's no advantage (~2-5 s, =
mlx-lm) and decode is ~19 t/s (no raw-speed gain). This is same-prompt reuse, not
cross-USER (which oMLX doesn't do); cross-restart persistence is documented but
not yet A/B-tested.

**TurboQuant KV compression** (llama.cpp). K=`q8_0` V=`turbo2` cuts KV RAM ~**28%**
(22.9 → 16.4 GB on a 4B model, M4 Pro) with tool-call validity unchanged (10/10),
and gains **+9% aggregate at 4-parallel** despite −8% single-stream. The symmetric
K=`turbo3` V=`turbo3` reaches ~−56% RAM but degrades quality (early-stop,
repetition) — the asymmetric `q8_0`/`turbo2` is the usable config.

---

## Section 4 — Memory & resources (Apple Silicon M5 Max 128 GB)

| # | Couple | Working-set RAM (GB) | Disk footprint (GB) | Swap Δ idle | Swap Δ under load | SOLO required? | Cohabitation safe? |
|---|--------|---|---|---|---|---|---|
| 1 | Rapid-MLX + Qwopus 35B-A3B-v1 | ~22 | 19.9 (MLX-4bit) | +0 | **+0 MB** | ⚠️ SOLO (cohabit thrash to 0.4 t/s) | ❌ |
| 2 | Rapid-MLX + Qwen 35B-A3B-UD | ~24 | 20.0 (MLX-4bit) | +0 | **+0 MB** | ⚠️ SOLO | ❌ |
| 3 | LM Studio + Qwen 35B-A3B-MTP | 21.6 | 23.2 (Q4 + MTP heads) | +0 | **+0 MB** | not tested | not tested |
| 4 | LM Studio + Qwopus 35B-A3B-v1 | 18.5 | 19.9 (Q4_K_S) | +0 | **+0 MB** | not tested | not tested |
| 5 | llama.cpp + Qwen 3.6-35B-A3B (reference) | ~16 | ~16 (Q5_K_XL) | +0 | **+0 MB** | ❌ | ✅ with `--parallel 2/3` |

> **"Under load"** = the 8-phase agentic bench including a 50K-token prefill (the
> heaviest *sequential* memory stress measured), M5 Max 128 GB, SOLO: swap delta
> **0 MB / 0 swapouts for every engine** — model + KV fit in free/inactive memory
> with >100 GB headroom. This is sequential-load memory, **not** 60-concurrent
> memory (see Section 2). Working-set RAM is an estimate; measured RSS includes
> mmap'd GGUF / wired MLX pages, so the true incremental footprint is lower (the
> MTP head adds ~+3 GB).

### Observations

- **Rapid-MLX requires SOLO operation on the GPU**: cohabitation with another
  actively-decoding engine triggers a swap delta of 5.4 → 14.2 GB and a decode
  collapse to 0.4 t/s. Do not start a second engine on the same Apple Silicon
  GPU.
- **LM Studio MTP** disk footprint is +13 % vs Q4_K_S without MTP heads, due to
  the MTP weight blocks. Negligible cost relative to the +17 % decode gain.
- On M5 Max 128 GB unified memory: every 35B-A3B configuration tested leaves
  more than 100 GB headroom after load — RAM is not the limiting factor.
- On M4 Pro 64 GB: `Q5_K_XL` does **not** fit alongside auxiliary models (swap
  thrash observed in production). `Q4_K_S` does fit.

---

## Section 5 — Model quality

> Public-benchmark figures here are **vendor / self-reported** and aggregated by
> leaderboards (llm-stats), not independently verified. Cross-validate at
> [llm-stats](https://llm-stats.com) · [LiveBench](https://livebench.ai) ·
> [SWE-bench](https://swebench.com) before relying on them. asiai's own direct
> measurements on Apple Silicon are in the next section.
>
> Author-only claims (Jackrong/Qwopus, Unsloth self-eval) are flagged separately
> and kept out of the public-leaderboard columns.
>
> 🔴 **Critical finding**: the "Hessling agentic" benchmark cited on several
> community model cards is **not independently reproducible** — 16 prompts,
> single curator, no neutral leaderboard integration. All three advisors
> recommend treating it as a smoke test only.

### Open-weight Qwen 3.6 base models

> Public-leaderboard figures (llm-stats), self-reported. The 27B-dense outscores
> the 35B-A3B MoE on SWE-bench — consistent with asiai's own dev-quality finding
> below (the MoE base is the one that hits the tool-call empty-object bug). MTP
> heads are a decode-speed feature and do not change a model's quality scores.

| Model | Architecture | SWE-bench Verified | GPQA Diamond | MMLU-Pro | Terminal-Bench 2.0 | BFCL |
|-------|--------------|-------------------:|-------------:|---------:|-------------------:|------|
| Qwen 3.6-35B-A3B-Instruct | MoE 35B / 3B active | 73.4% | 86.0% | 85.2% | 24.6% | absent from board |
| Qwen 3.6-27B-Dense Instruct | Dense 27B hybrid | 77.2% | 87.8% | 86.2% | 59.3% (vendor) | absent from board |

> Terminal-Bench **2.0** is far harder than the older Terminal-Bench v1 (community
> cards quote ~51.5% for the 35B-A3B on v1); the 24.6% here is the 2.0 generation.

### Qwopus 3.6 family — author-reported only, **not independently verified**

The Qwopus 3.6 finetunes published by Jackrong on HuggingFace claim
substantial gains over the Qwen base. As of May 2026 these claims have
**not been independently reproduced** on neutral leaderboards. Treat as
experimental until BFCL / SWE-bench reruns by a third party are
available.

| Model (author claims) | MMLU-Pro | SWE-bench Verified | Hessling agentic (16 prompts) |
|-----------------------|---------:|-------------------:|------------------------------:|
| Qwopus 3.6-35B-A3B-v1 (Jackrong) | claimed 88+ | claimed 75+ | claimed 88.6 ⚠ non-reproducible |
| Qwopus 3.6-27B-v2 (Jackrong) | claimed 87.43 | claimed 75.25 | n/a |

⚠ The "Hessling agentic" benchmark cited on the Jackrong model cards
appears to be a 16-prompt curator-specific evaluation with no neutral
leaderboard integration. All three advisories queried (Grok-4, GPT-5,
Gemini Advanced) recommend treating it as smoke test only.

### Frontier anchors (mid-2026)

> All figures are **vendor / self-reported**, aggregated by llm-stats — none are
> independently verified there. **Terminal-Bench 2.0** is the exception (the
> tbench team re-runs submissions; rows are peak agent×model scores). GPQA are
> vendor "Diamond" figures and the set is near-saturated — treat as approximate.

| Model | SWE-bench Verified | GPQA Diamond | MMLU-Pro | Terminal-Bench 2.0 | Source |
|-------|-------------------:|-------------:|---------:|-------------------:|--------|
| Claude Opus 4.8 | 88.6% | 93.6% | n/a | — (no TB submission) | llm-stats / Anthropic |
| Claude Opus 4.7 | 87.6% | 94.2% | n/a | **90.2%** | llm-stats / tbench |
| Claude Sonnet 4.6 | 79.6% | 89.9% | n/a | 53.4% | llm-stats / tbench |
| GPT-5.5 | n/a\* (SWE-Pro 58.6%) | 93.6% | n/a | 84.7% | OpenAI / tbench |
| GPT-5 (base) | 74.9% | 85.7% | n/a | 49.6% | llm-stats / tbench |
| Gemini 3.1 Pro | 80.6% | ~94.4% | n/a | 80.2% | llm-stats / tbench |
| DeepSeek-V4-Pro-Max | 80.6% | 90.1% | 87.5% | n/a | vendor (DeepSeek) |
| Llama-3.3-70B-Instruct | n/a | n/a | 68.9% | n/a | Meta (baseline) |

\* GPT-5.5 has no public SWE-bench *Verified* score (OpenAI reports SWE-bench Pro
Public 58.6%); the "88.7% SWE-bench" figure circulating is not on any primary
source. Note: **Qwen 3.6 has no 235B-A22B** — the open family is the 27B-dense
and 35B-A3B (below); the 235B-A22B is the prior Qwen3 generation.

### Same-class open-weights baselines

| Model | MMLU-Pro | SWE-bench Verified | Notes |
|-------|---------:|-------------------:|-------|
| Llama-3.3-70B-Instruct | ~75-80 | ~40-50 | Older but well-characterized baseline |
| Mistral Codestral 25.05 / Devstral | high (coding-specialized) | medium-high | Strong editor-style completion fidelity, weaker on reasoning |
| GLM-4.6-Coder (Zhipu) | vendor claims very high | disputed | Significant skepticism around evaluation methodology (consensus) |

### Quality benchmarks deprecated for this decision

- **HumanEval / HumanEval+** — saturated in 2026, all frontier models above 90 %, no signal left.
- **GSM8K** — saturated, no signal for coding agents.
- **MMLU (original)** — superseded by MMLU-Pro.
- **Author-reported "Hessling agentic" 16-prompt** — non-reproducible, treat as smoke test only.

### Open quality questions (research gaps)

1. **Quality-per-GB-RAM benchmark**: no standard exists. Proposed proxy formula:
   `AgentScorePerGB = (0.5·SWE + 0.3·BFCL + 0.2·TerminalBench) / RAM_resident`.
2. **Long-horizon stability (60+ tool calls)**: closest existing benchmarks are
   τ-bench, PencilPuzzleBench (>1000 turns), MultiAgentBench, TRAIL. None of
   them specifically measure "schema correctness and strategic coherence across
   60-80 sequential tool calls" — that benchmark gap is acknowledged by all
   three advisors.
3. **Conversion-aware evaluation (MLX-4bit vs GGUF Q4_K_M vs Q5_K_XL)**: no
   standardized leaderboard. Community reports diverge — some claim MLX-4bit
   preserves tool-calling stability worse than GGUF Q5_K_M, others say the
   opposite. **Practical advice**: run your own production workload against
   each quant before committing.
4. **Qwopus 3.6 family quality validation**: needs third-party BFCL +
   SWE-bench reruns. Author claims should not drive production decisions.

---

## asiai direct measurements — Apple Silicon, mid-2026

> What the public leaderboards above don't show: measurements asiai ran directly
> on Apple Silicon (M5 Max 128 GB in High Power Mode, M4 Pro 64 GB), llama.cpp
> b9430, deterministic (temp 0), on the public Qwen 3.6 family and the
> Opus-distilled **Qwopus** finetune. Caveat: cross-session absolute throughput on
> the M5 laptop is ±15% (thermal/load); only the **intra-session ±MTP back-to-back
> deltas** are tight, and M5↔M4 absolutes aren't comparable (different quants).

### Dev-quality / tool-call (`asiai bench --code`)

- The **base Qwen 3.6-35B-A3B (MoE)** collapses `edit_file.edits` to an empty
  object on the deep-context turn — **3/3 runs, at both Q4_K_S and Q5_K_XL**, same
  chat template. Tool-call clean **87.5%**, edit-turns clean **66.7%**. It is the
  MoE base's tool-call generation behaviour, not the quant and not the template.
- The **dense 27B** (Q5_K_XL) and **Qwopus-35B-A3B** (Q4_K_S) both score **100%
  clean / 0 bugs** — Qwopus reaches dense-27B tool-call reliability at the MoE's
  ~4× decode rate.
- Under a harder tool-call stress suite, Qwopus stays **100% / 0** while the dense
  27B drops to **88.9% / 3 bugs** (the same empty-object failure). But on an
  expression-evaluator trap (precedence of `**` vs unary minus) the **dense 27B is
  correct and Qwopus is wrong** — they split. (Recovery rate is weight-sensitive
  and noisy — not a headline.)

### Thinking ablation (`asiai bench --thinking-ablation`, Qwopus-35B-A3B, 3 deterministic runs)

| Config | Tool-call clean | Note |
|--------|----------------:|------|
| `enable_thinking=off` | **100%** | the only fully-clean config |
| `enable_thinking=on` + `preserve_thinking=on` | 77.8% | 2/9 turns dirty |
| `enable_thinking=on` + `preserve_thinking=off` | 11.1% | turns 2-8 → HTTP 500 (context corruption); avoid |

### MTP throughput (`--spec-type draft-mtp`, warm decode, intra-session ±MTP)

| Model / hardware | MTP off | MTP on | Δ |
|------------------|--------:|-------:|--:|
| 35B-A3B base · M5 Max | 85.5 t/s | **118.4 t/s** | **+38%** |
| Qwopus 35B-A3B · M5 Max | 105.7 t/s | 123.3 t/s | +17% |
| 27B-dense · M5 Max | 23.8 t/s | 28.0 t/s | +18% |
| Qwopus 27B · M5 Max | 25.9 t/s | 26.7 t/s | +3% |
| 35B-A3B MoE · M4 Pro | 36.3 t/s | 44.6 t/s | +23% |
| 27B-dense · M4 Pro | 10.4 t/s | 9.7 t/s | **−6%** |

MTP gain scales as **(MoE > dense) × (M5 > M4)** — strongly positive on the MoE,
marginal-to-negative on the slow dense path (the draft overhead isn't amortised).
The **Qwopus finetune's MTP head is also weaker than the base** (Qwopus 27B +3% /
35B +17%, vs base 27B-dense +18% / 35B-A3B +38%) — finetuning erodes the draft head.
The MLX-side MTP (mlx_vlm) is disqualified: it breaks long context (empty output,
75% valid). Headline: the 35B-A3B MoE + MTP on llama.cpp sustains **~118 t/s**
decode on M5 Max (~44 t/s on M4 Pro), ~4× the 27B-dense, at ~1.5 tok/s/W, TTFT
~62 ms, 100% output validity.

### Instruction-following (`asiai bench --instruct`, research-brief)

The thinking trade-off has teeth on multi-step deliverables: with
`enable_thinking=false`, Qwopus-35B does the tool work but delivers the requested
multi-section brief **0%** of the time (it stops at the secondary step); with
thinking on, the base model delivers it **100%** (5/5 sections). This pulls the
opposite way from the tool-call result above — thinking-off is cleanest for atomic
tool calls but suppresses written deliverables — which is why asiai sets thinking
**per task-dimension**, not as one global switch.

### Perfectionist research loop (`asiai bench --instruct loop-search`)

Single-turn IFEval and research-brief saturate at 100% across these models, so
neither surfaces the *perfectionist research loop*: a model that won't accept an
ambiguous, unconfirmable search result and re-issues semantically-equivalent
queries until a no-progress guardrail halts it, never delivering. A `loop-search`
sweep (9 configs, M5, b9430, thinking on/off, two ambiguity modes) isolates it:

- The **35B-A3B MoE loops to the cap** — for **both the base and the Qwopus
  finetune, in Q4 and Q8 alike**. The higher quant does not fix it, so the loop is
  **architectural to the A3B MoE**, not a quant artefact.
- The **dense 27B never loops** (Q4 / Q5 / Q8): it accepts the ambiguous result
  and writes the briefing.

So the throughput leader (the MoE, ~118-123 t/s) and the agentic-fitness leader
(the dense 27B, ~25 t/s) are *different models*. For a harness such as
NousResearch's Hermes Agent, loop-resistance can outweigh raw decode — the fastest
model is not always the right agent. (This is the inverse of the tool-call result,
where the MoE finetune was the more robust agent: **fitness is per-failure-mode,
so measure several.**)

---

## Section 6 — Operational

> 📌 Capability snapshot (mid-2026). Engine versions churn weekly on Apple
> Silicon — these cells are point-in-time, not a version-pinned guarantee.

| # | Engine | License | Stream OAI-compat | `/v1/models` | `/health` | `/metrics` (Prometheus) | Tool calling | Auto-DL HF | Persisted prefix cache | Maintainer activity |
|---|--------|---------|---|---|---|---|---|---|---|---|
| 1 | Rapid-MLX 0.6.66 | Apache-2.0 | ✅ | ✅ | ✅ (HTML page) | ❌ (logs only) | ✅ | ✅ HF Hub auto-DL on serve | ✅ `~/.cache/vllm-mlx/prefix_cache/` | community (raullenchai) |
| 2 | LM Studio 0.4.14 | proprietary | ✅ | ✅ | partial (websocket) | ❌ | ✅ | ✅ via `lms get` CLI | ❌ | Element Labs |
| 3 | llama.cpp b9270 | MIT | ✅ | ✅ | ✅ | ✅ `--metrics` | ✅ | manual (GGUF on disk) | ❌ (`--cache-reuse N` arch-disabled on hybrid) | ggerganov very active |
| 4 | mlx-lm | MIT | ✅ | ✅ | ✅ | ❌ | partial | ✅ HF auto | ❌ | Apple ml-explore active |
| 5 | oMLX | MIT | ✅ | ✅ | ✅ | ❌ | ✅ (caveat: post-cache-hit bug) | ✅ | partial (tiered SSD) | jundot active |
| 6 | vLLM-MLX | Apache-2.0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ paged-attention | vllm-project active |
| 7 | vMLX (Mamba/SSM) | Apache-2.0 | ✅ | ✅ | ✅ | partial | untested | partial | untested | community |
| 8 | Ollama | MIT | ✅ | partial | ✅ `/api/version` | ❌ | partial | ✅ `ollama pull` | ❌ | Ollama Inc. very active |

---

## Section 7 — Quality benchmark weighting for agentic-coding workloads

> This is the **asiai default weighting** for an orchestrator-class workload
> (60-80 sequential tool calls per turn, schema-validated output, long-context
> system prompts). It is informed by three frontier-LLM advisories
> (Grok-4, GPT-5, Gemini Advanced) queried May 2026, but is **not a community
> consensus** — treat as a starting point, not authoritative. Override via
> a future `--weights` flag (planned).

| Benchmark | What it measures | Why it matters here | Consensus weight |
|-----------|------------------|---------------------|-----------------:|
| **SWE-bench Verified** | Real GitHub repo navigation + patch + test repair | Best proxy for code-editing fidelity inside an agent loop | **35 %** |
| **BFCL v3** (Berkeley Function Calling Leaderboard) | Multi-turn function-call accuracy, argument fidelity, schema adherence | Direct predictor of orchestrator stability across many tool calls | **25 %** |
| **TerminalBench 2.0 / MCP-Atlas** | CLI and MCP task execution autonomy | "Does the agent survive 40+ actions without derailing" | **20 %** |
| **LiveBench Coding** | Contamination-resistant coding tasks (refreshed monthly) | Catches train-test leakage that inflates HumanEval-class scores | **10 %** |
| **Custom long-horizon stability eval** | 60-80 sequential tool calls with cumulative context growth, malformed JSON recovery | The benchmark that does not exist yet in public form — see Section 8 | **10 %** |

### Benchmarks consciously dropped from the weighting

- MMLU-Pro, GPQA Diamond, HumanEval+ — useful as a general capability signal,
  but **weakly correlated** with agent-loop reliability per 2026 evidence.
  Frontier-lab confirmations indicate single-shot reasoning scores no longer
  predict autonomous agent success at sufficient granularity.
- Author-reported aggregates without third-party reruns (Jackrong Hessling,
  Unsloth self-eval, GLM-4.6-Coder vendor claims).

---

## Section 8 — Custom "endurance" benchmark proposal (research opportunity)

All three advisors converge on the same gap: **the benchmark that would best
characterize an orchestrator workload does not exist publicly yet**. Building
one is the only way to get the missing signal.

### Proposed scope

- **80 sequential tool calls** per trajectory
- **Schema validation at every turn** (strict JSON / structured output)
- **Cumulative context growth** (10K → 50K tokens across the trajectory)
- **Interruption / recovery tests** (mid-trajectory cancel + resume)
- **Malformed XML/JSON recovery** (does the agent self-correct ?)
- **Repo-edit persistence** (do the edits made at turn N still hold at turn 60 ?)

This is on the asiai roadmap (a long-horizon endurance mode, after burst-mode).
If built, it would be the first public benchmark in this specific niche.

---

## Methodology

- **Hardware**: MacBook Pro M5 Max 128 GB unified memory, macOS 26.4.1.
- **Workload**: orchestrator class — system prompt ~7 KB, user prompt ~150-200
  tokens, 60-80 calls per turn.
- **Phases measured** (single-call, agentic-mode v1.6.0):
  - `cold`: first call after fresh start
  - `warm`: same exact prompt as cold (warm cache)
  - `prefix-test-1/2/3`: identical system, user changing — measures cross-USER cache reuse
  - `cold-prefix`: identical system, after restart — measures persistent cache
- **Verdict prefix cache reuse**: `YES` if `median(prefix-test) / cold < 0.2`,
  else `NO`.
- **Anti-bias measures**: SOLO mode (no cohabiting engines), thermal idle
  baseline, mmap warm-up phase.
- **Quality gates** (auto-tracked by asiai bench):
  - `early_stop`: at least 2 runs with `<0.5×` median completion
  - `memory_pressure`: swap delta `>500 MB` OR swapouts delta `>1000`
  - `duplicate_processes`: multiple engine processes detected during the bench

The full protocol is the `asiai bench --agentic-mode` / `--burst-mode`
instrumentation (power/thermal, engine footprint, KV occupancy, prefix-cache
phases) — see the asiai CLI docs.

---

## Open questions

1. **MTP on vLLM-MLX/Rapid-MLX — answered (partly).** vLLM-MLX added MTP in
   prerelease **0.4.0rc1** (2026-05-21); the theoretical combo "MLX + MTP-equipped
   Qwopus 35B-A3B + cross-USER snapshot" could win on both decode and TTFT once
   the Rapid-MLX fork tracks 0.4.x. Track when Rapid-MLX picks up the MTP path.
2. **MTP on the MLX runtime — current state.** Released mlx-lm does not run the
   MTP head as native speculative decoding (`sanitize()` drops the MTP weights
   during conversion; native support is in the unmerged PR
   [ml-explore/mlx-lm#990](https://github.com/ml-explore/mlx-lm/pull/990)).
   LM Studio's `mlx-engine` wraps mlx-lm, so it inherits this — the +13.5%
   decode gain in Section 1 row 5 comes from LM Studio's **llama.cpp-derived
   backend** (the file is GGUF), not from mlx-engine speculative decoding.
3. **Burst behavior on Rapid-MLX/vllm-mlx at 60-80 calls scale**: smoke
   test confirms single-slot FIFO at burst=5. Full panel pending (Section
   2). The relevant upstream issue is whether vllm-mlx plans
   continuous-batching / multi-slot scheduling for hybrid arch models.
4. **`llama_memory_can_shift=false` on Qwen 3.6 hybrid** — still broken
   upstream. [#18497](https://github.com/ggml-org/llama.cpp/issues/18497) is
   closed (documents full re-processing); [#22384](https://github.com/ggml-org/llama.cpp/issues/22384)
   is an *issue* (closed-as-completed), **not** a merged fix; the actual fix PR
   [#23121](https://github.com/ggml-org/llama.cpp/pull/23121) was **closed
   unmerged** (patches live only on forks). The "just enable `preserve_thinking`"
   workaround is refuted by open issue
   [#22615](https://github.com/ggml-org/llama.cpp/issues/22615) (0.67× speedup =
   cache stays inert). The hybrid DeltaNet layers don't expose a shiftable cache
   state by construction.
5. **Qwopus 3.6 quality independent reproduction**: needs third-party
   BFCL / SWE-bench reruns. Author-published numbers should not drive
   production decisions until cross-verified.
6. **vllm-mlx vs Rapid-MLX lineage — answered.** Rapid-MLX is a community
   **hard fork** of `waybarrios/vllm-mlx`, not a thin wrapper: it vendors the
   engine in-tree (package still named `vllm_mlx`), does not pip-depend on the
   upstream package, and has diverged substantially (Rapid-MLX 0.6.74 vs upstream
   0.3.0). The shared `vllm_mlx` package name and `~/.cache/vllm-mlx/` dir are a
   frequent source of attribution confusion (see Section 3, caveat 2).

---

*This panel is a living document. Contributions, corrections, and additional
bench cells welcome via [github.com/druide67/asiai](https://github.com/druide67/asiai/issues).*



---

## /research/comparison-panel/2026-06-qwen-agentworld-35b

Raw markdown: https://asiai.dev/markdown/research/comparison-panel/2026-06-qwen-agentworld-35b.md
Rendered: https://asiai.dev/research/comparison-panel/2026-06-qwen-agentworld-35b/

# Qwen-AgentWorld-35B on Apple Silicon: should it get a slot in your agent loop?

> An evaluation brief for people who run local models and build autonomous agents.
> **What it is**: a *language world-model* — it predicts what a terminal would
> output after an action, it does not act. **What runs**: MLX, or llama.cpp/Metal
> with a one-line metadata override (a plain GGUF won't load without it); no
> official MLX build. **Its one differentiator we measured**:
> it holds the simulator role across multi-step sequences where a generalist drifts.
> **Its cost**: heavy over-reasoning — cappable. Numbers are small-N and directional,
> each tagged with its sample size; author benchmark figures are flagged as claims.
>
> Measured with `asiai` on an M5 Max, MLX 4-bit, one engine at a time, 2026-06.
> Corrections welcome via [github.com/druide67/asiai](https://github.com/druide67/asiai/issues).

!!! tip "When to use it / when not"
    **Use it as** an environment simulator for cheap agent rollouts, a mock for
    tool/terminal output, or a trajectory verifier in place of an LLM-as-judge
    (*the verifier use case is untested here — see §6*). It also holds up as a
    plain 35B generalist if you prompt it as an assistant.

    **Don't use it as** your daily assistant: the authors ship no chat/code usage
    path and it carries a steep over-reasoning tax (cappable, see §5). And don't
    wait for the 397B variant that "beats GPT-5.4" — it is **not downloadable**
    (HF returns 401 despite the Apache-2.0 announcement).

## 1. Runnability & reproduction (read this first)

If it doesn't run on your machine, nothing else matters. Verdict, blunt:

- **Two paths work today; neither is turnkey.** There is **no official MLX build** —
  we used a community MLX conversion, and that is the path we measured on. The GGUF
  **also loads** on llama.cpp / Metal, but not out of the box: as-is it fails with
  `missing tensor 'blk.40.attn_norm.weight'` (build 9780, re-confirmed 2026-06-25).
  The cause is a converter off-by-one, **not missing weights** — the GGUF declares
  `block_count=41` (an extra MTP layer at index 40) while shipping only the 40 real
  layers 0–39, so llama.cpp asks for a layer that was never meant to exist. Override
  the metadata at load and it loads *and generates*:
  `--override-kv qwen35moe.block_count=int:40 --override-kv qwen35moe.nextn_predict_layers=int:0`.
  Ollama and LM Studio wrap llama.cpp but don't reliably expose `--override-kv`, so
  treat those two as untested. Official server deployment is vLLM / SGLang / Transformers.
- **A quant that loads is not proof it emits a correct long chain-of-thought** —
  validate generation, not just load.

Reproduction setup:

| | Repo (Hugging Face) | Size |
|---|---|---|
| AgentWorld (specialist) | `jedisct1/Qwen-AgentWorld-35B-A3B-oQ4-MLX` | ~20 GB |
| Qwen3.6 (generalist baseline) | `mlx-community/Qwen3.6-35B-A3B-4bit` | ~19 GB |

`mlx-lm` 0.31.3 · M5 Max 128 GB · sampling temp 0.6 / top-p 0.95 / top-k 20 · one model loaded at a time.

!!! warning "Token budget is a first-class setup variable"
    AgentWorld emits a very long reasoning trace. At `max_tokens=4096` its output
    is **truncated before the answer** and scores as a false failure. It needs
    **8192–12288** reasoning tokens to finish on some trivial cases. Anyone
    re-running at a low budget will get worse-looking numbers for AgentWorld that
    are harness artifacts, not model errors.

**RAM / context fit**: weights ~20 GB; peak ~27 GB at 64K context on a 128 GB Mac;
the KV cache grows only ~5 GB from 4K to 64K (a property of the shared hybrid
architecture). A 64 GB Mac runs it comfortably at reduced context; 36–48 GB is
tight but workable at 4K–32K.

## 2. What it is, and how the authors position it

A **language world-model**: given a state and an action (a typed command), it
predicts the next observation (what the terminal returns) via a long
chain-of-thought. Seven digital domains (MCP, Search, Terminal, SWE, Android, Web,
OS). It is trained to *be the environment*, not to act in it.

The authors ship it **as a world-model, not an assistant**: the system prompts are
simulation prompts, and there is no documented chat/code usage path. So a fair
worry is that, used as an assistant, it would simulate a console output instead of
answering. Our test nuances this (§4): with a standard assistant prompt it codes
and reasons on par with the generalist. **The behavior is decided by the prompt,
not by a lost capability.**

!!! note "On the word *world-model*"
    The most common community objection is terminological: this is an
    autoregressive LLM doing next-text-state prediction, not a non-autoregressive /
    energy-based world-model in the LeCun sense. Worth knowing before the name sets
    an expectation the model doesn't claim to meet.

Verified specs (HF model card, in-the-clear):

| | |
|---|---|
| Parameters | **34.66 B** total · ~3 B active (MoE) |
| Architecture | `qwen3_5_moe`, hybrid **Attention + Gated-DeltaNet** |
| Experts | 256 (8 routed + 1 shared) |
| Context | up to **256K** tokens |
| License | **Apache-2.0** (~65 GB in BF16) |

## 3. The differentiator: multi-step role fidelity

This is the one new, defensible result — and exactly what the authors' own
benchmark never measures (it is single-step only). The test: chain commands that
build state (create a dir, enter it, write a file, read it back) and, at each step,
have the model predict the exact terminal output.

Frame it as a **reliability** property — format/role discipline — **not** a
comprehension advantage. Qwen3.6 understands the terminal perfectly well (it tracks
the working directory, counts the right lines); the difference is that it sometimes
*leaves the role*.

| Test | AgentWorld | Qwen3.6 | Note |
|---|---|---|---|
| Plausible output (`ls`, `git`, `ps`) — N=3 | 9/9 | 9/9 | parity |
| Sequence A — 6 steps, anchored (4 runs) | 0 role-breaks / 24 steps | intermittent | role-hold |
| Sequence B — 8 steps, anchored (3 runs) | 0 role-breaks / 24 steps | intermittent | role-hold |
| Closed-loop (feeds itself) — N=2 | 6/6 ×2 | intermittent | role-hold |

**Honest reading**: AgentWorld broke role in **0 of 48 observed steps** across two
sequences and four runs. Qwen3.6 breaks role intermittently — its anchored runs
swung 0/6 → 6/6 across repeats (N=2), so this is **directional, not a rate**. When
it fails, it **regurgitates the action JSON** instead of simulating the output:

```text
$ cat log.txt              # log.txt was just deleted → env must return an error

AgentWorld (in role):
  root@host:/home/user# cat log.txt
  cat: log.txt: No such file or directory
  root@host:/home/user#

Qwen3.6 (out of role, ~1 run in 2 here):
  [{"keystrokes": "cat log.txt\n", "duration": 0.1}]    # echoes the input command
                                                        # instead of the output
```

The correct answer is often present in Qwen3.6's output — it is a **format/role**
failure, not a misunderstanding. For a loop where each step must be machine-readable
by the next, a single role-break poisons the chain, which is what AgentWorld avoids.

!!! note "Measurement caveats (disclosed)"
    Byte-exact scoring on the command-echo line is strict, and our Sequence-D vs
    Sequence-E fixtures were inconsistent about whether a `cd` observation includes
    the echo — so the role-fidelity metric has a known wrinkle. The direction is
    robust across four files; the precise gap is not.

## 4. Generalist capability: the base is not degraded

The owner's question (did the world-model fine-tune break the base LLM?) gets one
sober section, not the headline. Short answer: no — N=3, directional.

| Task | AgentWorld | Qwen3.6 | |
|---|---|---|---|
| Reasoning (5 verifiable puzzles incl. the strawberry-'r' trap) | 15/15 | 15/15 | parity |
| Code generation (4 functions, **executed against unit tests**) | 12/12 | 12/12 | parity |

Run with an assistant prompt (not the simulator prompt), AgentWorld writes correct
code and reasons correctly, at parity with the generalist. It does not "derail" — it
is a competent generalist that happens to over-reason.

## 5. The cost: an over-reasoning tax — and the remedy

Promote this from a footnote to an adoption gate, because for a per-step verifier it
is the deciding number — but it has a fix.

Measured on deterministic terminal cases (N=2 per case):

| Mode | AgentWorld | Qwen3.6 |
|---|---|---|
| Reasoning **on** (default simulator mode) | median **1140 tok/pred**, max 2558 · ~14 s · 8/8 exact | 504 tok · ~4.5 s · 8/8 |
| Reasoning **off** (`enable_thinking=false`) | **45 tok/pred · ~0.5 s · 8/8 exact** | 45 tok · ~0.4 s · 8/8 |

AgentWorld emits ~2.3× more tokens than the generalist and on a trivial `cd ; pwd`
its reasoning ran **past 8192 tokens in 2 of 3 runs**. The final answer is correct —
this is a latency/compute tax per step, not a correctness defect.

!!! tip "The remedy: cap it"
    Turning reasoning **off** for the simulator role cuts tokens ~25× and latency
    ~28× **with no loss of byte-exact fidelity** on deterministic cases (still 8/8).
    For a per-step verifier or mock, run it with `enable_thinking=false` and a
    `max_tokens` ceiling. **Caveat**: this is tested on deterministic cases only —
    on outputs where the reasoning genuinely helps (ambiguous state, complex
    content), reasoning-off may cost fidelity. Untested here.

## 6. Performance (single-run, indicative ★)

Same family, same architecture, so the profiles are close. Read these as trends.

| Measure | AgentWorld | Qwen3.6 | Reading |
|---|---|---|---|
| Time to first token ★ | ~360 ms | ~510 ms | AW ahead |
| Decode throughput ★ | ~110 t/s | ~117 t/s | ~7% slower |
| Decode at 64K context | ~132 t/s | ~160 t/s | ~73% retained |
| Memory 4K → 64K | +5 GB | +5 GB | hybrid arch, not AW-specific |
| Context cache (13K-token prefix reuse) | ~×21 | ~×23 | **MLX property**, not the model |

The ~7% decode gap is most likely the 4-bit recipe (AgentWorld protects its
linear-attention projection in 6-bit; Qwen3.6 protects the MoE gate in 8-bit), on
unequal output lengths — a confound, not a model disadvantage. Prompt caching is an
mlx-lm feature identical on both models; its ~20× gain scales with the cached prefix
length, it is not a property of AgentWorld.

**Untested but high-value (the community's #2 use case)**: using next-state
prediction as a *trajectory verifier* — when the real environment diverges from the
prediction, that signals an off-path agent. We did not measure its false-positive /
false-negative behavior. Open question.

## 7. What the authors claim

!!! quote "Author benchmark — a claim, not a measurement"
    On their own benchmark (AgentWorldBench), AgentWorld-35B scores **56.4**, level
    with Claude Sonnet 4.6 (56.0). The gains they attribute to specialization, by
    ablation against the **base Qwen3.5** (self-reported, not a head-to-head vs
    Qwen3.6): **+21.9** tool-use (MCP), **+18.1** software engineering, **+10.2**
    terminal. Thesis: *world-model specialization beats generational improvement* —
    the generalist Qwen3.6 scores **below** the base (42.9 vs 47.7) on simulation
    fidelity, because it is tuned to *act*, not to *predict state*.

    These figures come from a single-source, in-house benchmark graded by an LLM
    judge, on a model less than 48 h old at publication — **no third-party
    replication**. The top of their table sits within ~2 points under one judge, so
    near-the-top ordering is within noise; the 397B "beats GPT-5.4" margin is +0.46
    (noise), and that variant is non-public (HF 401) despite the Apache-2.0
    announcement.

Our multi-step result (§3) is on a *different, non-replicated metric* than their
single-step bench; it points the same direction (Qwen3.6 weaker at simulation), but
that is thesis convergence, not confirmation.

## 8. How I'd wire it in

- **Prompt**: use the official terminal **simulation** system prompt to run it as an
  environment; use a plain assistant prompt only if you want generalist output. The
  two modes are different jobs.
- **Cost control**: `enable_thinking=false` + a `max_tokens` ceiling for the
  simulator role (§5). With reasoning on, budget ~1000–2500 tokens/step.
- **Closed loop**: feed back the model's own predictions, but anchor on the real
  environment when you have it; expect format strictness to matter (the echo line).
- **Footprint**: ~20 GB weights, ~27 GB peak at 64K.
- **The build-vs-adopt question**: is "never leaves role" intrinsic to the
  world-model training, or could a generalist + grammar-constrained decoding close
  most of the gap? We did not test the constrained-generalist alternative — weigh it
  before adopting a dedicated model.

## Limits of this bench

- **Small samples** (N=1–5, no standard deviation). Every numeric gap is a trend,
  not a statistical result.
- **One domain** for the two key results (terminal sequences). Role-hold "in a loop"
  remains to be confirmed elsewhere.
- **Quantization not isolated**: the two 4-bit recipes differ slightly; the decode
  gap is likely tied to that but it is not proven here.
- **Not yet tested**: random/complex scenarios, a second domain, a three-way against
  the base Qwen3.5 to isolate the fine-tune's exact effect, and the trajectory-verifier
  use case.
- **Only the 35B is public.** The 397B variant is not downloadable.

---

*Sources: arXiv 2606.24597 · [Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) (Apache-2.0). Results internally cross-reviewed for bias before publication. ★ = single, indicative measurement.*



---

## /turboquant

Raw markdown: https://asiai.dev/markdown/turboquant.md
Rendered: https://asiai.dev/turboquant/

---
title: "TurboQuant Benchmark on Apple Silicon: Run 70B Models on Mac"
description: "Real benchmarks of TurboQuant KV cache compression on Mac Mini M4 Pro 64GB: Llama 70B at 6.3 tok/s with 5x memory savings. Setup guide and results."
type: article
date: 2026-03-31
updated: 2026-03-31
faq:
  - q: "Can I run a 70B model on a Mac with 64GB RAM?"
    a: "Yes, with TurboQuant. The KV cache is compressed 5x, so Llama 70B Q4_K_M (40GB weights) fits comfortably in 64GB with 32K context. We measured 6.3 tok/s on a Mac Mini M4 Pro."
  - q: "Does TurboQuant reduce quality?"
    a: "No measurable quality loss. The perplexity increase is under 1% vs q8_0, and Needle-in-a-Haystack retrieval scores 100% through 32K context."
  - q: "Which TurboQuant format should I use?"
    a: "We recommend asymmetric: q8_0 for keys (sensitive to compression) and turbo3 for values (5x compression, no quality impact). This is based on findings from the turboquant_plus project."
  - q: "Does TurboQuant work with MLX engines?"
    a: "Community MLX implementations exist but are less mature than the llama.cpp fork. For production use on Apple Silicon, we recommend TheTom/llama-cpp-turboquant with Metal kernels."
  - q: "How much faster is TurboQuant?"
    a: "Decode speed is about 0.9x of q8_0 (slightly slower per token), but prefill can be faster at long context due to reduced memory bandwidth. The real gain is fitting larger models and longer contexts in the same RAM."
---

# TurboQuant Benchmark on Apple Silicon

TurboQuant (Google Research, ICLR 2026) compresses the KV cache of LLMs by 5x with no quality loss, enabling 70B models to run on a Mac Mini with 64GB RAM. These are real benchmarks measured with [asiai](/) on actual hardware.

## Results

**Llama-3.1-70B-Instruct Q4_K_M on Mac Mini M4 Pro 64GB**

| Metric | Value |
|--------|-------|
| **Throughput** | 6.3 tok/s (stable, CI 95%: 6.3-6.3) |
| **TTFT** | 196 ms (median) |
| **GPU Power** | 23.8 W |
| **Model VRAM** | 44.1 GB (40 GB weights + 4 GB KV turbo3) |
| **Context** | 32,768 tokens |
| **GPU Offload** | 81/81 layers on Metal |
| **Thermal** | Nominal (no throttling) |
| **Stability** | Stable (std dev 0.04 tok/s across 3 runs) |

KV cache configuration: keys at q8_0 (high precision), values at turbo3 (3-bit, 5x compression).

## Before vs After TurboQuant

| | Without TurboQuant | With TurboQuant (turbo3) |
|--|-------------------|--------------------------|
| **KV cache (32K ctx)** | ~20 GB (q8_0) | ~4 GB (turbo3) |
| **Total RAM needed** | 60+ GB (OOM on 64GB) | 44 GB (fits in 64GB) |
| **Can run 70B on 64GB?** | No | **Yes** |
| **Quality** | Baseline | -1% PPL (negligible) |
| **NIAH retrieval** | 100% | 100% |

## What Is TurboQuant?

TurboQuant is a KV cache compression algorithm from Google Research, presented at ICLR 2026. During LLM inference, the KV cache stores intermediate attention states and grows linearly with context length. For a 70B model at 128K context in FP16, this cache alone can consume 20-40 GB of RAM.

TurboQuant compresses this cache to 3 bits per value using:

- **Random rotation** (Walsh-Hadamard transform) to Gaussianize the data
- **Optimal scalar quantization** (PolarQuant) near the Shannon limit
- **QJL** (Quantized Johnson-Lindenstrauss) to preserve dot products

The result: 5x memory reduction, no fine-tuning needed, and near-zero quality loss.

## Setup Guide

### Hardware

- Mac Mini M4 Pro, 64 GB unified memory ($2,700)
- Any Apple Silicon Mac with 32+ GB should work (adjust model size accordingly)

### Install TurboQuant llama.cpp

```bash
# Install build tools
brew install cmake

# Clone the TurboQuant fork
git clone https://github.com/TheTom/llama-cpp-turboquant.git
cd llama-cpp-turboquant
git checkout feature/turboquant-kv-cache

# Build with Metal (Apple Silicon GPU)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(sysctl -n hw.ncpu)
```

### Download a Model

```bash
# Llama 3.1 70B Q4_K_M (~40 GB)
curl -L -o llama-3.1-70b-q4_k_m.gguf \
  "https://huggingface.co/bartowski/Meta-Llama-3.1-70B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf"
```

### Raise macOS GPU Memory Limit

```bash
sudo sysctl iogpu.wired_limit_mb=61440
```

### Launch the Server

```bash
./build/bin/llama-server \
  -m llama-3.1-70b-q4_k_m.gguf \
  --cache-type-k q8_0 --cache-type-v turbo3 \
  -c 32768 \
  --port 8081 \
  --host 0.0.0.0 \
  -fa 1 \
  -ngl 99 \
  -t 10 \
  --no-mmap \
  --chat-template chatml
```

### Configuration Explained

| Parameter | Value | Why |
|-----------|-------|-----|
| `--cache-type-k q8_0` | Keys at 8-bit | Keys are sensitive to compression |
| `--cache-type-v turbo3` | Values at 3-bit | Values tolerate extreme compression (5x) |
| `-fa 1` | Flash Attention | Required for TurboQuant |
| `-ngl 99` | Full GPU offload | All 81 layers on Metal |
| `-t 10` | 10 threads | M4 Pro has 10 performance cores |
| `--no-mmap` | No memory mapping | Loads everything at boot, avoids page faults |
| `--chat-template chatml` | ChatML format | Best compatibility with this fork |

## Benchmark with asiai

```bash
pip install asiai
asiai detect --url http://localhost:8081
asiai bench --engines llamacpp --prompts code --runs 3 --kv-cache turbo3 --card
```

## Models That Fit on 64GB with TurboQuant

| Model | Weights (Q4_K_M) | KV Cache (32K, turbo3) | Total | Status |
|-------|-------------------|----------------------|-------|--------|
| Llama 3.1 70B | 40 GB | ~4 GB | 44 GB | **Tested: 6.3 tok/s** |
| Qwen2.5 72B | 40 GB | ~4 GB | 44 GB | Should work |
| Llama 70B 128K ctx | 40 GB | ~16 GB (turbo3) | 56 GB | Tight but possible |
| Command-R+ 104B | 58 GB | ~4 GB | 62 GB | Very tight |

## FAQ

**Can I run a 70B model on a Mac with 64GB RAM?**

Yes, with TurboQuant. The KV cache is compressed 5x, so Llama 70B Q4_K_M (40GB weights) fits comfortably in 64GB with 32K context. We measured 6.3 tok/s on a Mac Mini M4 Pro.

**Does TurboQuant reduce quality?**

No measurable quality loss. The perplexity increase is under 1% vs q8_0, and Needle-in-a-Haystack retrieval scores 100% through 32K context.

**Which TurboQuant format should I use?**

Asymmetric: q8_0 for keys + turbo3 for values. Keys are sensitive to compression (all quality degradation comes from K compression). Values can be compressed to 2-3 bits with zero effect on attention quality.

**Does TurboQuant work with MLX?**

Community implementations exist ([turboquant-mlx](https://github.com/helgklaizar/turboquant_mlx)) but are less mature than the llama.cpp fork. For production use, we recommend [TheTom/llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant).

**How does this compare to standard llama.cpp?**

Decode speed is ~0.9x of q8_0 (slightly slower per token), but the real gain is fitting models and contexts that simply didn't fit before. Prefill can actually be faster at long context due to reduced memory bandwidth pressure.

## References

- [Google Research Blog — TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/)
- [TurboQuant Paper (ICLR 2026)](https://arxiv.org/abs/2504.19874)
- [TheTom/turboquant_plus](https://github.com/TheTom/turboquant_plus) — Extended implementation with Sparse V
- [TheTom/llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant) — llama.cpp fork with Metal kernels
- [llama.cpp Discussion #20969](https://github.com/ggml-org/llama.cpp/discussions/20969) — Community thread



---

## /versions-mode

Raw markdown: https://asiai.dev/markdown/versions-mode.md
Rendered: https://asiai.dev/versions-mode/

# Versions — running vs installed vs available

`asiai versions` answers a question single-host tools usually can't:
**is the engine I have the latest one, and is the process I'm running the
one I have installed?** It lines up three coordinates per engine:

- **running** — the version of the live engine process (read from its HTTP
  endpoint or a `--version` shell-out, the same detection `asiai detect`
  uses).
- **installed** — what is on the machine (Homebrew formula/cask, a pip
  package, or a macOS app bundle).
- **available** — the latest upstream version. Offline by default
  (`brew outdated` against the local Homebrew cache); PyPI/GitHub when you
  pass `--check-upstream`.

From those it derives a status:

| Status | Meaning |
|--------|---------|
| `up-to-date` | installed == available (or no upstream signal and running == installed) |
| `upgrade-available` | a newer version exists upstream |
| `running-stale` | the **running process predates the installed binary** — you upgraded but didn't restart. A restart reconciles it. |
| `not-installed` | nothing installed and nothing running |
| `unknown` | a version string couldn't be parsed, or there's nothing to compare against |

## CLI

```sh
asiai versions                      # offline: running/installed + brew outdated
asiai versions --check-upstream     # also query PyPI / GitHub (network, opt-in)
asiai versions --engine llamacpp    # filter to one engine
asiai versions --json | jq          # machine-readable
```

Example:

```
Engine versions

  ENGINE     RUNNING  INSTALLED  AVAILABLE  STATUS
  ─────────  ───────  ─────────  ─────────  ─────────────────
  llama.cpp  9370     9370       9380       upgrade-available
  Ollama     —        0.24.0     0.24.0     up-to-date
  Rapid-MLX  —        0.6.68     0.6.68     up-to-date

  llama.cpp: https://github.com/ggml-org/llama.cpp/releases
  Ollama:    https://github.com/ollama/ollama/releases

  AVAILABLE is brew-cache only (offline). Pass --check-upstream for PyPI/GitHub.
  1 upgrade(s) available
```

A `running-stale` row is the classic post-upgrade trap: `brew upgrade
llama.cpp` bumped the binary, but the `llama-server` process you started
last week is still the old build. The fix is `aisctl restart llamacpp`
(or whatever your engine is), not another upgrade.

## Web dashboard

`asiai web` gains a **/versions** page: the same three-column table with
status badges and clickable changelog links, auto-refreshed via HTMX.

```sh
asiai web                                # http://127.0.0.1:8899
open http://127.0.0.1:8899/versions
open http://127.0.0.1:8899/versions?upstream=1   # include PyPI/GitHub
```

The JSON API mirrors the CLI:

```sh
curl -s localhost:8899/api/v1/versions | jq
curl -s 'localhost:8899/api/v1/versions?upstream=1' | jq
```

Results are cached (60 s offline, 10 min for the network mode) so opening
the page doesn't hammer brew/PyPI/GitHub on every refresh.

## Doctor recap

`asiai doctor` runs an **offline** version recap under the Engine section
so a single `doctor` pass tells you if anything is behind:

```
  Engine
    ⚠ Versions             1 upgrade(s): llama.cpp
      Fix: asiai versions
```

It never makes a network call (only the local `brew outdated` plus the
reachability probes doctor already does), so `doctor` stays fast.

## Where the engine list comes from

`asiai` ships an internal table mapping each engine to its Homebrew
formula / pip package / GitHub repo, so the feature works standalone. When
[`asiai-inference-server`](https://github.com/druide67/asiai-inference-server)
is installed alongside `asiai`, it contributes a richer table through the
`asiai.version_sources` entry point — adding the engines it manages
(e.g. `turboquant`) and the authoritative formula mapping. The two
packages never import each other; the provider hands over plain data and
`asiai` merges it **field by field over its internal defaults** (a
provider that knows the brew formula but not the app-bundle path won't
erase asiai's own fallback). Rows sourced from the provider are labelled
accordingly in `--json` (`"source": "aisrv"`).

## Upgrading

Reading versions is read-only by design. Triggering an upgrade is a
**write** and lives in `asiai-inference-server`:

```sh
aisctl upgrade llamacpp --restart   # brew upgrade (formula-whitelisted) + restart
aisctl upgrade llamacpp --dry-run   # show the brew argv without running it
aisctl upgrade llamacpp             # upgrade only; prints a hint to restart later
```

Without `--restart`, the binary is upgraded but the running daemon keeps
executing the old build (the `running-stale` state) — useful when you don't
want to interrupt an in-flight request. Restart on your own schedule with
`aisctl restart llamacpp`.

The `/versions` web page **does not** trigger upgrades itself. It renders
copy-paste CLI snippets for the engines that need attention, mirroring the
fleet dashboard's deliberate choice to keep write actions out of the
browser: a live POST button would mean an operator's Bearer token lives in
`localStorage`, where a stored XSS in any version/notes field could steal
it and ride the authenticated upgrade endpoint. The snippet approach keeps
the token in your shell only.

For cross-host upgrades, the authenticated `asiai web → aisctl serve` fleet
surface already supports `upgrade`:

```sh
aisctl fleet push studio upgrade --engine llamacpp
```

See [fleet-mode.md](fleet-mode.md) for that security model (Bearer token,
per-token rate limit, audit log).

## Caveats

- **`brew outdated` reflects the local cache.** The `available` column is
  only as fresh as your last `brew update`. `asiai versions` never runs
  `brew update` for you (it's slow and mutates state) — run it yourself
  before relying on the offline column.
- **GitHub rate-limits unauthenticated requests** to 60/hour per IP. Set
  `GITHUB_TOKEN` in the environment to lift that to 5000/hour; otherwise
  `--check-upstream` degrades gracefully to `available: —` with a note
  rather than failing.
- **App-bundle engines have no public upstream API** (e.g. LM Studio).
  Their `available` column stays empty; the status is derived from
  running-vs-installed only.
- **pip lookups use the environment `asiai` runs under**
  (`sys.executable -m pip`), which can differ from the venv your engine
  runs in. The reported pip version is asiai's view.
- **mlx-lm can be installed via both brew and pip.** Brew takes
  precedence for the `installed` column, matching `doctor` and the
  upgrade path.