Running a local AI model smoothly on a $214 used office machine

Can you run a local AI model smoothly on a used office machine that costs about as much as one year of a chatbot subscription? I was skeptical. Then I tried it, and the machine keeps surprising me. This post is the full setup with real numbers, including the parts that are not impressive.

The machine

A used mini PC with an i5-10400T: 6 cores, 35 W TDP, no GPU, SATA SSD. I paid 10,500 pesos for the box and 2,650 pesos for 2x16 GB of DDR4, about $214 total at the time of writing.

Fair warning: that was a hunted deal. RAM prices have gone up since, so the same build takes some searching now. But the class of machine matters more than the exact deal: off-lease office computers with 6 cores and two RAM slots are everywhere, and they are the cheapest path to 32 GB of memory, which is the real requirement here.

Why this works at all: MoE

The trick is not the hardware, it is the model architecture. Dense models this size crawl on a CPU. Mixture-of-experts models changed that: Qwen3-30B-A3B has 30 billion parameters, but only about 3 billion are active for any given token. You get close to small-model speed with big-model answers. The whole model must sit in RAM, which is why 32 GB is the entry ticket, but the compute per token stays modest.

First attempt: Qwen3-30B-A3B

My first runs with llama.cpp gave about 4 tokens/s. After tuning thread counts I got it to roughly 5.3 tokens/s, with prompt processing around 26 to 29 tokens/s. Usable for chat, noticeably slow for long answers. Promising enough that I kept tweaking.

The upgrade that surprised me

Months later I updated llama.cpp and downloaded Qwen3.6-35B-A3B (Q4_K_M quant, 22 GB). The difference was bigger than I expected:

• generation: ~5.3 to ~7.9 tokens/s
• prompt processing: ~29 to ~40 tokens/s
• and the model is larger (35B vs 30B) and a generation newer

One honesty note: the 30B numbers were measured on the older llama.cpp build, so that comparison mixes model improvements with runtime improvements. I have not re-benchmarked the 30B on the current build yet. Either way, the practical outcome stands: the bigger, smarter model now runs about 50% faster on the same box than the older one ever did.

The stack around it

Everything is open and self-hosted on that one machine:

• llama.cpp server with its web UI
• Qwen3.6-35B-A3B Q4_K_M pinned in RAM
• SearXNG for private web search
• sqlite-vec for RAG over my own docs
• nimblegate, my git push gateway (more on that below)

The web UI reaches every tool over MCP. That is the part that makes a small box feel smart: a 35 W CPU model cannot know things, but it can fetch them. Web search brings sources, RAG brings my notes, and the git gateway brings live data about my own repositories.

Asking my infrastructure about itself

The gateway is the fun part. Every push I or my coding agents make goes through a self-hosted gate that checks the diff against exact rules and records every decision. The gate exposes its decision log over MCP too, so the assistant can query it.

I asked: “how did pushes to the nimblegate repo do this month?”

It called the gate’s stats tool and answered in 41 seconds: 99 decisions, 98 accepted, 1 rejected, plus which rules keep firing. The AI that writes the code can ask what the gate thought of it. The audit trail is queryable by the thing being audited.

The follow-up was better: “which findings keep coming back?” Nine rules ride about 90 of those 99 pushes. An AWS documentation-example key in a migration guide. PEM keys in test fixtures. A curl-piped-to-shell line in the installer. All intentional content, and the gate correctly flags its own examples, the same way antivirus test files trip antivirus. My favorite: the whitelist file trips the secrets rule, because it contains the very patterns it whitelists. That is not noise. That is a to-do list.

(That repo runs in observe mode, where findings are recorded but not blocked, exactly because the tool’s own repo is worst-case by construction. Enforce, where findings block the push, is the default for normal repos.)

The warm-start trick

Cold model load used to make the first question sluggish. The fix: the service pre-warms the model’s KV cache at boot with the system prefix, 2,825 tokens of it. The first message after a restart processed in 1.6 seconds. The box does the waiting, not me.

RAM budget, honestly

• 32 GB total
• ~22 GB pinned for the model
• ~29 GB peak with the vision encoder, desktop, and browser all running
• so about 3 GB of headroom at peak; normal chat leaves more

It runs alongside daily apps fine. When I need memory back, the older 30B frees 4 GB, and a small 4B model frees about 20 GB in seconds.

What you actually get

• ~7.9 tokens/s generation, ~40 tokens/s prompt processing
• a web-searched, source-grounded answer in about 30 seconds end to end
• CPU around 50 C while generating
• $214 of used hardware, 35 W at the wall

Not datacenter speed. But it is mine, it runs offline, and every number above came off this machine, not from a spec sheet.

The gate in the middle of it is nimblegate, source-available and self-hosted, if you also run AI agents against repos you care about.