Every prompt you send to a cloud chatbot leaves your house. For most questions that’s fine. But the moment you want to paste in proprietary code, summarize personal documents, or just tinker without a subscription, the cloud starts to feel like the wrong place for the job.
The good news: in 2026, the small open models finally got good enough that a genuinely useful chatbot fits on hardware you probably already own. Google’s Gemma 4 family runs from a 2-billion-parameter edge model up to a 31B workstation model, and OpenAI’s open-weight gpt-oss brings real reasoning and tool-calling to a 20B model that fits in 16GB of memory.
What “running it locally” actually means
A local LLM is just a file of model weights sitting on your disk, plus a program that runs it. You download the weights once, a free tool like Ollama or LM Studio loads them into memory, and from then on the conversation happens entirely on your machine. No account, no API key, no per-token bill — and your prompts never leave your network.
The trick is quantization: weights stored at 4 bits per parameter instead of 16, shrinking a model to roughly a quarter of its full size. The rule of thumb is about 0.6GB of memory per billion parameters — so an 8B model needs roughly 5GB, and a 27B model roughly 18GB. Memory, not CPU speed, decides what you can run.
The memory ladder: what fits where
| What you have | What runs well | First model to try |
|---|---|---|
| 8GB RAM (old laptop, small mini PC) | Small chat and summarization | gemma3:1b or gemma4:e2b |
| 16GB RAM (base Mac mini, decent laptop) | Capable daily-driver chat, vision, light coding | gemma4:12b, gpt-oss:20b |
| 24–32GB RAM (upgraded mini, old gaming PC) | Strong reasoning, coding agents, big context | gemma4:26b, gpt-oss:120b |
Start small on purpose: a tiny model that answers in two seconds beats a giant one that crawls. The smallest models are genuinely tiny — smollm2:135m is a 271MB download — perfect for proving the setup works before you commit gigabytes.
Up and running in ten minutes
Ollama is the one-line install: a download from ollama.com/download on Mac, or curl -fsSL https://ollama.com/install.sh | sh on Linux. Then:
ollama run gemma4:12b
That’s it — the model downloads and you’re chatting. Ollama also serves an OpenAI-compatible API: ollama launch claude --model gpt-oss wires Claude Code to a local model, so your coding assistant reads your repo without it ever leaving your disk.
Prefer a window to a terminal? LM Studio is the friendly option: a free desktop app that browses, downloads, and chats with models in a proper GUI, with the same local API at localhost:1234.
Hardware you already own
Apple’s current Mac mini with M6 starts at $899 with 16GB of unified memory — enough for the 12B-class models — and the 32GB configuration runs $1,299, opening up the 27B tier. It sips power, too: Apple lists the base M6 mini at around 4W at idle, so leaving it on 24/7 costs almost nothing. (Here’s a tagged Amazon search for the Mac mini M6.)
An old gaming PC with a mid-range GPU works just as well — VRAM is memory too, and a 12GB or 16GB card runs the same 12B–20B models. A cheap mini PC with 16GB of DDR5 handles the small models fine.
One practical note: model files are big, and the base Mac mini ships with only 256GB of storage. A handful of models will eat that fast — plan on an external SSD for your model library, far cheaper than Apple’s storage upgrades.
What it’s actually good for
Set expectations honestly: a local 12B or 20B model is not GPT-5 or Claude. It’s a very capable junior assistant — excellent at drafting, summarizing, and explaining, weaker at the hardest reasoning. What it is good for: private drafting that never touches a server, summarizing your own documents without uploading them, coding help without leaking the repo, and offline use on a plane or during an outage — all with zero API bill.
The one thing to get right: don’t expose it
Here’s the catch. Ollama’s local API listens on 127.0.0.1:11434 by default and has no built-in authentication. That’s fine on your own machine — but rebind it to 0.0.0.0 and anyone who can reach that port can run prompts on your GPU and pull new models.
This isn’t theoretical. Cisco Talos counted 1,139 Ollama servers reachable from the public internet in a September 2025 scan, 214 of them actively serving models to anyone who asked. And in October 2026, Symantec caught an autonomous “Skynet” agent in a honeypot that hijacks exposed AI endpoints and makes the owner pay the inference bill. Keep your server on your LAN, reach it from outside only over a VPN like Tailscale or an SSH tunnel, and never forward port 11434 from your router.
The bottom line
A private AI on your own hardware used to be a datacenter project. Now it’s a weekend project: one free download, one command, and a model that answers in seconds without phoning home. Start small on the machine already on your desk, and upgrade your memory — not your subscription — when you want more.
Sources & further reading: Ollama model library — Gemma 4 · Ollama model library — gpt-oss · Ollama model library — Gemma 3 · LM Studio · Mac mini M6 at $899 (Mac Observer) · Exposed Ollama servers and the Skynet LLMjacking report · Running small models locally with Ollama · How to run a local LLM at home
As an Amazon Associate I earn from qualifying purchases, which helps cover the cost of keeping this site online.