Running a Local AI on Your Business Laptop — What It Actually Takes

When people talk about running AI locally, they usually mean one of two things: they tried it once and got frustrated, or they work in ML and have a workstation that cost $8,000. The practical middle ground — a business professional who just wants AI triage and drafting on a normal laptop — is underserved.

This is that guide.

Why Bother With Local AI?

Three real reasons:

Privacy. Your email content, client names, financial figures, and internal documents never leave your machine. With cloud AI (GPT-4, Claude), every prompt you send is transmitted to and temporarily held by a third party. For routine consumer tasks this is fine. For business communications, it's a liability.

Cost at scale. API costs are low per query but compound fast. If you're running AI triage on 200 emails/day, that's ~$15–40/month depending on model and token usage. Local inference is a one-time GPU investment or simply uses compute you already own.

Offline capability. No internet? AI still works. For email triage on a flight, drafting in a hotel with bad Wi-Fi, or simply not depending on an API that has occasional outages — local is more reliable.

The Hardware Reality

Here's what the marketing copy doesn't say clearly: the bottleneck is GPU VRAM, not CPU speed.

Large language models run fastest when the entire model fits in GPU memory. When it doesn't, the system offloads layers to RAM and then to disk (in the worst case), and performance degrades sharply.

What you actually need by tier:

Minimum viable (4 GB VRAM): Any modern laptop with a discrete GPU. You can run Phi-3 Mini (3.8B parameters) fully on-GPU. Response latency ~2–4 tokens/second. Good enough for email triage; not great for long documents.

Comfortable (8 GB VRAM): GeForce RTX 3060/4060 or equivalent. Llama 3.1 8B fits entirely in VRAM. 12–20 tokens/second. This is the sweet spot for business productivity tasks. An RTX 4060 laptop GPU currently runs ~$900 in a mid-range laptop.

Excellent (16+ GB VRAM): RTX 4080/4090 or Mac M-series (unified memory). Runs 13B and 14B models fully on-GPU. 15–30 tokens/second on a 13B model. If you have an M2/M3/M4 MacBook Pro with 16+ GB RAM, you're already in this tier — Apple Silicon's unified memory architecture means the GPU and CPU share memory.

What about CPU-only?

You can run LLMs on CPU with llama.cpp. Expect 1–3 tokens/second on a modern laptop CPU. Usable for email triage (you don't watch it generate in real time anyway), but noticeably slow for interactive chat. If this is your only option, use a 3B–4B parameter model.

Which Model Should You Use?

The model landscape changes every few months. These are reliable choices as of mid-2026:

Llama 3.1 8B (Meta) — Best general-purpose model for 8 GB VRAM. Excellent instruction following, good at summarization and drafting. Pull with: ollama pull llama3.1:8b

Phi-3 Mini 3.8B (Microsoft) — Remarkably capable for its size. Fits in 4 GB VRAM. Best for tasks that are clearly structured (triage, classification, short drafts). Struggles with nuanced long-form writing. Pull with: ollama pull phi3:mini

Mistral 7B — Strong for European languages and coding tasks. If your business runs in French, German, or Spanish, Mistral often outperforms Llama at the same size. Pull with: ollama pull mistral:7b

Llama 3.1 70B (quantized to Q4) — If you have 32+ GB RAM or 16+ GB VRAM. This is competitive with GPT-4 on most business tasks. Don't attempt it on 8 GB VRAM.

Setting Up Ollama (5 Minutes)

Ollama is the simplest way to run local models. It handles downloads, quantization, GPU offloading, and serves a local API that applications can use.

1. Download from ollama.ai and install (Linux, macOS, Windows supported)

2. Pull your first model:

ollama pull llama3.1:8b
# Downloads ~4.9 GB on first run

3. Ollama starts automatically and listens on localhost:11434. Any application that supports the OpenAI-compatible API can use it.

4. Applications like JARVOS detect Ollama automatically — the status indicator in the sidebar changes to AI: Local as soon as Ollama is running.

Realistic Performance Expectations

Here's what to actually expect, so you're not disappointed:

Email triage (classify + 1-sentence summary): 0.5–2 seconds per email on an 8B model with GPU. This runs in the background — you never wait for it.

Draft reply (3 sentences): 4–8 seconds visible generation time. Feels fast enough for interactive use.

Summarise a 5-page PDF: 15–30 seconds. Worth it for a long document you'd otherwise spend 10 minutes reading.

Chat about a complex topic: Streaming at 10–20 tokens/second feels natural on 8 GB VRAM. Below 5 tokens/second starts to feel slow.

When to Fall Back to Cloud AI

Local AI is not always the right choice within a session. For heavy reasoning tasks — complex contract analysis, multi-step financial calculations, tasks that need GPT-4 quality — the cloud API produces significantly better output.

The right approach is a fallback chain:

• Use local AI by default (free, private, offline)
• Fall back to cloud when local isn't available or the task is complex
• Make the active AI path visible so you always know what's running

JARVOS implements this chain automatically: Ollama first, then llama-server, then Anthropic Claude, then OpenAI. The status indicator in the sidebar tells you which path is active at all times.

Bottom Line

If you have a laptop with 8 GB+ GPU VRAM, or a modern Mac with 16+ GB unified memory, you can run high-quality AI locally today. The setup takes 10 minutes. The ongoing cost is zero per query.

If you're on older hardware, local AI still works — just pick smaller models (Phi-3 Mini, Llama 3.2 3B) and set appropriate expectations for generation speed.

The combination that works best for a typical solopreneur: Ollama + Llama 3.1 8B for 90% of tasks, with an Anthropic API key as backup for the 10% that need more power.