Which Local AI Model Fits Your Hardware and Tasks?
Choose local AI hardware from your tasks, not parameter counts: memory budgets, quantisation, context and concurrency, plus a reproducible benchmark to run before buying anything.
Contents
- Start with tasks and acceptable performance
- Understand the memory budget
- Choose candidate models and configurations
- Benchmark the same tasks consistently
- Read quality and performance together
- Choose a configuration, or a different architecture
- Frequently asked questions
- Do I need a discrete GPU?
- Does fitting a model in memory mean it is useful for my tasks?
- How do context size and concurrency affect requirements?
- Can I use local retrieval with a cloud model?
Image: Vitaly Sacred on Unsplash.
Pick local AI hardware by testing your actual tasks on candidate models, not by matching parameter counts to a chart. A model that fits in memory is not necessarily good at your work, and a slower model that gets the answer right may cost less per useful result than a fast one that does not.
This guide gives you a method rather than a shopping list. Model releases and hardware prices move too quickly for a fixed recommendation to stay true, so you measure on your own machine and decide from your own numbers.
It is part of the AI harness engineering guide. For the privacy side of going local, see local vs cloud vs hybrid.
Start with tasks and acceptable performance
Write down three workload profiles before looking at hardware.
| Profile | Example | What matters most |
|---|---|---|
| Short Q&A | "When is the Alpha review?" from a few notes | Accuracy, a few seconds of latency |
| Document-heavy | Summarise twenty pages with citations | Context size, citation accuracy |
| Tool-using | Prepare a meeting brief using calendar and notes tools | Reliable tool calls over several steps |
For each, set a quality bar using a few test questions with known answers, and an acceptable wait. A two-minute wait may be fine for an overnight brief and unacceptable for a quick question.
Some workloads need the strongest models available. If testing shows local candidates cannot meet your quality bar, a hybrid setup that keeps sensitive tasks local and sends the rest to a cloud model may serve you better than more hardware.
Understand the memory budget
A local model needs memory for three things:
the model weights,
the runtime's own overhead,
the working memory for the context you give it.
Weights depend on the model size and how heavily it is quantised. Quantisation stores weights at lower precision, which shrinks memory use and often speeds things up, at some cost in quality that you should measure rather than assume.
Context memory grows with how much text you send and how many requests run at once. A model that loads comfortably with a short prompt can run out of memory with a long document or two simultaneous tasks.
Where the model runs matters too:
| Placement | Typical effect |
|---|---|
| Fully on a GPU with enough VRAM | Fastest |
| Split between GPU and system RAM | Works, slower |
| CPU and system RAM only | Works for smaller models, much slower |
| Apple silicon unified memory | GPU and CPU share one pool |
Check how your runtime reports placement and context. Ollama's FAQ, for example, shows ollama ps, whose Processor column reads 100% GPU, 100% CPU or a split such as 48%/52% CPU/GPU. At the time of writing it also documents a default context window of 4,096 tokens, changeable with OLLAMA_CONTEXT_LENGTH, and a default bind address of 127.0.0.1 port 11434 (Ollama FAQ). Test with the context length you will actually use, not the default. If you want to be sure nothing goes to the cloud, the same FAQ documents a local-only mode, OLLAMA_NO_CLOUD=1, that disables its cloud models and web search.
Choose candidate models and configurations
Shortlist three to five candidates and record, for each:
| Field | Why |
|---|---|
| Model name and exact version | Results do not transfer between versions |
| Quantisation level | Changes memory, speed and quality |
| Context length you will use | Changes memory and sometimes quality |
| Tool-calling support | Required for the tool-using profile |
| Licence and permitted use | Check it yourself for your intended use |
Leave any field you have not verified marked as unverified. Model licences in particular vary and change; read the actual licence for the version you download.
Benchmark the same tasks consistently
Use the same questions, documents, prompts and limits for every candidate.
- Prepare ten tasks per profile with expected answers and sources.
- Fix the settings: prompt, context length, temperature, step and time limits.
- Warm up each model once, then record cold-start time separately from normal runs.
- Run each task three times per configuration.
- Record: pass or fail against your criteria, time to answer, peak memory, and any errors or crashes.
| Configuration | Short Q&A passed | Docs passed | Tools passed | Median time | Peak memory | Notes |
|---|---|---|---|---|---|---|
| Candidate A, setting 1 | ?/30 | ?/30 | ?/30 | ? | ? | |
| Candidate A, setting 2 | ?/30 | ?/30 | ?/30 | ? | ? | |
| Candidate B, setting 1 | ?/30 | ?/30 | ?/30 | ? | ? |
Fill the table from your own runs. The evaluation guide explains how to grade fairly and why repeated runs matter.
Read quality and performance together
Speed without accuracy is a fast way to be wrong. Read each row as a pair:
- Fast and failing: reject, however quick.
- Accurate and slow: acceptable for background tasks such as overnight briefs.
- Accurate on short tasks, failing on long documents: check evidence placement and context size. A 2023 study found that the models of the time often did best when the relevant passage was at the start or end of a long input and worse in the middle (Liu et al.). Test whether that applies to your candidates rather than assuming it.
- Failing on tool use only: consider a hybrid route for tool-heavy tasks.
Then convert to cost per successful task with the cost worksheet, including electricity and the share of hardware cost.
Choose a configuration, or a different architecture
| Your results | Reasonable choice |
|---|---|
| A candidate passes all three profiles on current hardware | Use it; no purchase needed |
| Passes short and document tasks, struggles with tools | Local for reading, cloud or hybrid for tool-heavy work |
| Passes only with more memory than you have | Price the upgrade against hybrid and cloud scenarios |
| Nothing local meets the quality bar | Cloud or hybrid, with sensitive data kept out |
Whatever you choose, record the date, versions and results. Rerun the benchmark when you change models or upgrade hardware. For the wider trend towards running models at home, see the rise of local AI, and for software options, the best local AI agents.
Frequently asked questions
Do I need a discrete GPU?
Not always. Smaller models run on CPU and system memory, and some laptops share memory between CPU and GPU. Whether that is fast enough depends on your tasks. Test before buying.
Does fitting a model in memory mean it is useful for my tasks?
No. Fit is the minimum. It must also pass your quality bar on your own tasks.
How do context size and concurrency affect requirements?
Both increase memory use. Test with your longest realistic documents and the number of tasks you expect to run at once.
Can I use local retrieval with a cloud model?
Yes. Many hybrid setups keep documents and search local and send only selected passages to a cloud model. Decide which data classes may leave the machine first.