Windhover

Why it exists

7B on a 16 GB Mac shouldn’t thrash.

Stock fp16 chat models are memory-bandwidth bound. When RSS climbs, macOS swaps — and tokens per second collapse. Windhover keeps a sparse working set in RAM so real models stay usable under a hard ceiling.

Weights alone are huge ~14 GB for a typical 7B in fp16 — before KV and activations.
Decode streams bytes every token If every step touches a dense footprint, you blow past a healthy budget.
Residency, not just quantization Only the active slice competes for RAM. Cold weights stay mmap’d on SSD.

~780×

Faster decode vs stock transformers on Qwen2.5-7B

4.19 GB

Peak RSS with Windhover — under a ≤ 9 GB budget

9.80

Decode tok/s on MacBook Air M4 · 16 GB

Benchmarks

Measured on a MacBook Air M4.

Same model. Same machine. ≤ 9 GB process budget with swap-abort enabled. Not projected. Not cloud.

Model: Qwen/Qwen2.5-7B-Instruct · Host: Apple M4 · 16 GB · Date: 2026-07-22

Without Windhover

stock transformers · CPU · fp16

Decode 0.013 tok/s

Peak RSS 8.82 GB

Fits ≤ 9 GB? borderline

Status swap-bound

With Windhover

windhover-engine · KPK

Decode 9.80 tok/s

Peak / mean RSS 4.19 GB

Fits ≤ 9 GB? yes

Status ok

Without

0.013 tok/s

Windhover

9.80 tok/s

Qwen2.5-7B — with vs without Windhover

MacBook Air M4 · 16 GB · ≤9 GB budget · decode-only · measured locally

~780×

Decode throughput (tok/s)

0.013

Without transformers · fp16

9.80

Windhover windhover-engine · KPK

Resident set (GB) 9 GB cap

8.82

Without swap-bound

4.19

Windhover fits budget

	Without	With Windhover	Δ
Decode	0.013 tok/s	9.80 tok/s	~780×
RSS	8.82 GB (thrash)	4.19 GB	−53%
Usable chat on 16 GB Mac	no	yes	—

Host pressure

After without-engine swap stress, Windhover was re-run on the same machine. RSS stays flat. Throughput drops under thermal / memory pressure — honest, not hidden.

Condition	Decode	RSS
Quiet (earlier same day)	9.80 tok/s	4.19 GB
After without-engine swap stress	5.62 tok/s	4.19 GB

Host pressure — same Mac, same day

Windhover re-run after without-engine swap stress · RSS stays flat

Quiet (earlier)

9.80 tok/s

4.19 GB RSS

Before stock-path thrash

After without-engine swap stress

5.62 tok/s

4.19 GB RSS

~42% slower · memory still capped

Bench notes

Decode-only where available · greedy · Windhover NGEN=32 · Stock path: CPU fp16 via transformers · Budget ≤ 9 GB with swap abort. A live without-engine attempt was aborted at 6.44 GB RSS after swap jumped +1.5 GB; the 0.013 tok/s figure is from an earlier run allowed to finish under thrash. Full dumps live in the docs/ folder.

How it works

Sparse working set. Hard ceiling.

Models convert to KPK packs. Activation units (AUs) sit under one byte ledger. Hot stays resident. Cold stays on SSD.

01

Install

Pull a pack from Library. Mac 16 GB models download real HF weights.

02

Convert

Build a KPK — group-int4 weights, arch descriptor, gate metadata.

03

Run

Chat and Agent talk to windhover-engine under a RAM ceiling.

04

Watch

Advanced shows live RSS, tok/s, footprint, sparsity, AU hit.

Activation units

Dense models (Qwen, Llama, …) use FFN neuron bundles gated by magnitude (CATS-style). MoE models treat routed experts as AUs. Kernels only touch the predicted bytes for the next token.

Why this beats “just quantize”

Quantization helps, but a naive load still wants a large contiguous working set. Windhover’s point is residency control — a 7B stays near ~4 GB RSS instead of thrashing toward 9 GB+.

┌──────────────┐     gate / router      ┌─────────────────┐
│  KPK pack    │ ─────────────────────▶ │  AU tiers       │
│  group-int4  │                        │  hot · warm ·   │
│  mmap weights│                        │  cold (SSD)     │
└──────────────┘                        └────────┬────────┘
                                                 │
                                                 ▼
                                        ┌─────────────────┐
                                        │ RAM ledger      │
                                        │ hard ceiling    │
                                        │ (RAM_GB / cap)  │
                                        └────────┬────────┘
                                                 │
                                                 ▼
                                        bandwidth-first
                                        CPU kernels

Features

Everything you need. Nothing fake.

The Mac and Windows apps wrap Library · Chat · Agent · Advanced around windhover-engine.

Hard RAM ceiling

Process stays inside a budget (RAM_GB / hard-cap ledger).

KPK + mmap

Group-int4 packs; cold weights stay on SSD instead of a full fp16 heap.

Sparse AUs

Dense FFN bundles and MoE experts under one working-set policy.

Library

Browse, install, and uninstall local packs. Honest catalog — no fake stubs.

Chat

Markdown streaming UI; only chat-capable installs appear.

Agent

Folder-scoped list / read / write on device — lightweight local coding agent.

Advanced

Live telemetry: RSS, latency, tok/s, sparsity, AU hit — no silent model swap.

Inside the app

Library. Chat. Agent. Advanced.

Four surfaces. One engine. Clear status at every step.

Browse Mac-16 GB packs and larger models. Install what you actually have disk for.

Local replies with speed and RSS chips per message.

Pick a folder. The local model lists, reads, and writes under that root only.

Live engine telemetry — RSS, tok/s, Windhover decode stats.

Download

Get Windhover.

macOS Apple Silicon and Windows 11 (x64 + ARM64). Always the latest GitHub Release asset.

macOS

Apple Silicon

DMG for M1–M4. Drag Windhover into Applications.

Download .dmg → Windows 11

x64

NSIS installer for Intel / AMD. WebView2 usually preinstalled.

Download .exe → Windows 11

ARM64

For Snapdragon and other ARM64 PCs.

Download .exe →

First launch: builds are currently unsigned. On macOS, right-click → Open, or run xattr -cr /Applications/Windhover.app. On Windows, More info → Run anyway. Requires macOS 12+ (Apple Silicon recommended) or Windows 11.

Browse all assets on GitHub Releases. The repo may still be named Kestrel; the product is Windhover.

Docs

Quick start & CLI.

Prefer the app for day-to-day use. The CLI is there when you want the engine bare.

From the app

Install a pack from Library → convert to KPK when prompted → open Chat or Agent. Advanced shows live RSS and tok/s. Models live under ~/.windhover/models (or %USERPROFILE%\.windhover\models).

Requirements

macOS 12+ · Apple Silicon recommended · Windows 11 x64/ARM64 · From source: Xcode CLT, Rust, Node 18+, Python 3.10+ with torch + transformers for convert.

CLI

./windhover build
./windhover doctor
./windhover pull Qwen/Qwen2.5-7B-Instruct --weights
./windhover convert ~/.windhover/models/Qwen__Qwen2.5-7B-Instruct
./windhover app                 # http://127.0.0.1:8000

./windhover chat --model ~/.windhover/models/Qwen__Qwen2.5-7B-Instruct/kpk \
  --prompt "Hello" --ngen 64

./windhover bench --windhover

Source on GitHub Install notes Full README Upstream / license Engine docs

Devlog

What’s shipping.

Recent milestones from the Windhover engine and desktop apps.

Jul 21

Windows ARM64 release assets v0.1+

Fixed Windows ARM64 engine arch and reattached full release assets. NSIS installers for x64 and ARM64 ship beside the macOS DMG.
Jul 21

Flat geometric icon

Refreshed the Windhover mark to a flat geometric bird on deep green — same copper accent used across the product.
Jul 20

Windows 11 app support

Tauri desktop for Windows 11 with embedded windhover-server + windhover-engine. No repo checkout or MinGW required for end users.
Jul 19

Agent coding demo & honest benches

Folder-scoped Agent surface, refreshed screenshots, and typed bench tables with clean SVG charts — with vs without Windhover on M4 Air.
Jul 18

Windhover sparse engine ships

Product rebrand from Kestrel to Windhover. Dense KPK path with CATS-style sparsity, hard RAM ledger, and measured M4 benches under a ≤ 9 GB ceiling.

Local models that fit your machine.