Running a local model stopped being a hobby project

The first time most people try to run a model locally, they download something far too large for their machine and watch everything freeze. They conclude local AI is not ready. It is a reasonable conclusion from the evidence, and it is out of date.

What actually changed

Not the models, mostly. Small open models were already decent. What changed is that the setup stopped being a project:

• Quantised weights mean a genuinely useful model fits in a few gigabytes rather than tens
• Runtimes handle the hardware detection you used to do by hand
• The gap between a 7B model and a frontier one closed enough that, for summarising a document or drafting a reply, you often cannot tell

The remaining problem is not capability. It is that nothing tells you which model your computer can actually run before you spend twenty minutes downloading one.

The question worth answering first

How much memory do you have free, right now, and which models fit in it comfortably rather than technically?

That is a boring question and it is the one that decides whether your first attempt works. We put it at the front: the catalogue reads your available memory and marks each model comfortable, tight, or out of reach before you commit to the download.

Eight gigabytes of RAM runs a small model fine. Sixteen is comfortable for mid-size. No GPU is required — it makes things faster, it is not a condition of entry.

What it is good for, honestly

Local models are excellent at the high-volume, low-glamour work: reading documents, extracting fields, classifying, summarising, drafting a first pass. That is most of what an automation actually does.

They are not the right tool for everything, and pretending otherwise is how people end up disappointed. For hard reasoning you will still reach for a frontier model — which is why connecting your own API key sits right next to the local option rather than being hidden.

The difference is that once the weights are on your disk, running them costs nothing per use. That changes what is worth automating: things you would never send to a metered API become obvious once each run is free.

The Models module is free forever, including on the free tier. There is a longer write-up on the private LLM runner page.