Your security team flags the new AI coding tool because every prompt your developers send carries source code out to a third-party cloud. If you work at a fintech, that single fact can disqualify the tool, and hospitals and defense contractors keep hitting the same wall. The work still has to ship, and the code still cannot leave the building.

That tension is why a local large language model (LLM) has moved from a hobbyist curiosity into a real engineering decision. If your team has strict data and compliance requirements, you are probably already looking at on-premise inference now that the models and tooling are good enough to trust. This guide covers what a local LLM is, why you would run one, the hardware and tools involved, and how to run one with a coding agent so a local model does real engineering work.

What a local LLM is and how it differs from a cloud API

A local LLM is a language model that performs inference on infrastructure you control, such as a laptop, GPU workstation, on-prem server, or private cloud instance. In a properly configured local setup, prompts and outputs do not need to leave your environment. The privacy boundary depends on the full stack: the model runtime, agent, tools, telemetry settings, retrieval sources, and any optional cloud connectors you enable.

If you are buying for compliance, keep local inference separate from private API tiers, because they are not the same guarantee. Inference on a "private" cloud plan still runs on someone else's servers, so a managed private plan does not satisfy a requirement to keep data inside your own infrastructure. Local inference can also reduce  network round trips, though real latency depends on hardware, model size, quantization, runtime, and context length.

The tradeoff is greater operational responsibility in exchange for greater control. When you self-host, model updates, GPU drivers, monitoring, and scaling all land on your team. Cloud APIs hand you the managed infrastructure and redundancy you would otherwise build yourself.

Why run an LLM locally

Most teams land on a local model for one of four reasons, and it helps to figure out which one actually settles your decision before you spend a dollar on hardware. These four map onto the constraints that regulated and high-volume teams keep hitting.

• Data privacy and sovereignty: When the model runs on your server, prompts, responses, and any documents in the workflow can stay inside your network when the model runtime, retrieval sources, tools, telemetry, and integrations are configured accordingly. For regulated industries, local inference can simplify compliance reviews because sensitive prompts, source code, or protected data do not need to be sent to an external model provider. Teams still need legal and security review, but the data-flow story is often easier when inference stays inside controlled infrastructure.

• Cost predictability for high-volume workloads: Cloud APIs charge per token, which turns expensive once volume climbs. Self-hosting front-loads the hardware and operations cost, then holds the marginal cost per call steady once the machine is running.

• Latency and no rate limits: Local inference keeps the model right next to the application, so your own hardware sets the performance ceiling instead of a third-party queue, a rate limit, or a network dependency. You feel this most on agentic runs that fire many calls per task.

• Less vendor lock-in: Tie your operations to one cloud provider and you inherit their pricing changes and feature deprecations, and switching later can force prompt rewrites and output-regression checks. Running open-weight models on infrastructure you control gives teams more flexibility to change runtimes, deployment environments, and model providers over time.

Whichever of these turns out to be your binding constraint, the next question is what machine can actually run the model you have in mind.

What hardware you need to run a local LLM

Memory governs almost every hardware choice you make here. A model's weights have to fit in video memory (VRAM), or in unified memory on Apple Silicon, and quantization is the lever you use to fit larger models on smaller machines. Quantization compresses weights from 16-bit floating point down to 4-bit or even 2-bit integers, so a 16-bit model in the tens of gigabytes can shrink substantially once you drop it to 4-bit. The GGUF format is the common currency across many local runtimes, and 4-bit is usually where you want to start on consumer hardware.

Model sizes vary by runtime, format, and context length, so treat the model file as a floor and add headroom for the key-value (KV) cache, activations, and framework buffers. Treat the tiers below as a rough starting map, then validate against your actual model and runtime.

Memory capacity is usually the wall you hit first. A 70B model at 4-bit generally needs more than a single 24 GB consumer GPU offers once you count runtime overhead, so you either offload, add GPUs, or move to Apple Silicon, whose unified memory sidesteps part of that constraint. You can still get real work done without a GPU, since CPU-only inference runs fine with small quantized models at slower speeds, where memory bandwidth is the bottleneck rather than raw core count.

The local LLM stack: runtimes, servers, and agent layers

The local LLM stack has three layers. Model families are the weights you run, such as Qwen, Gemma, DeepSeek, Devstral, or Phi. Inference runtimes and servers, such as Ollama, LM Studio, llama.cpp, Jan, GPT4All, and Lemonade Server, serve those models locally. Agent platforms like OpenHands use those model endpoints to perform software engineering work. Local inference determines where model execution happens. OpenHands provides the workflow layer that turns that model into a software engineering agent capable of working against repositories, running tests, and producing reviewable changes.

Running a model locally improves control over inference, but it doesn't automatically make an AI workflow private. Repository access, retrieval systems, external tools, telemetry, logging, credentials, and cloud integrations can all affect where data flows. Teams evaluating local deployments should review the entire execution path rather than assuming local inference alone satisfies their security requirements.

1. OpenHands

OpenHands is an open-source platform for building and running AI coding agents. It’s the agent layer for teams that want a local model to do real software engineering work. A runtime like Ollama, LM Studio, Lemonade Server, or llama.cpp serves the model. OpenHands uses that model as the inference backend for coding-agent workflows: reading a repository, editing files, running commands, testing changes, and producing reviewable outputs. Running a local LLM gives you private inference. Running OpenHands with a local LLM gives you an agent workflow that can use that private inference to work on code.

Key features:

• Agent layer over any runtime: OpenHands can use any supported local interface endpoint as the model backend for software engineering workflows, allowing locally hosted models to participate in coding-agent tasks.

• Open and self-hostable: The core is MIT-licensed, so teams can inspect, extend, and self-host the open-source agent framework. If the workflow uses hosted model providers, external tools, or enterprise-only components, those parts remain governed by their own terms and deployment boundaries.

• Enterprise control layer: OpenHands Enterprise adds governance capabilities such as access controls, auditability, usage visibility, budgets, and sandboxed runtimes for organizations running agents across teams.

Pros:

• Automates the full engineering loop rather than single completions, so a modest local model does real work.

• When paired with a local runtime and configured without external model calls, teams can keep agent execution and inference within the infrastructure they control.

Cons:

• It is an agent platform, not an inference runtime, so you still need a model server like Ollama underneath.

• The full control plane with budgets and scoped permissions lives in the paid Enterprise tier.

Best for: Teams that want a local model to run autonomous engineering tasks, and organizations that need to scale from one laptop agent to a governed fleet.

2. Ollama

Ask most developers where to start with local inference and they name Ollama first. It hides model downloads and local execution behind a single command-line interface (CLI), and it exposes an OpenAI-compatible endpoint the moment a model is running.

Key features:

• One-command models: Running ollama run gemma3:4b downloads and starts a model in a single step.

• OpenAI-compatible endpoint: It serves the API at http://localhost:11434/v1, so existing OpenAI clients point straight at it.

Pros:

• Almost no setup between install and a working local model.

• Wide model library with simple tag-based versioning.

Cons:

• The CLI-first design gives you less visual control than a graphical app.

• Fine-grained serving parameters are less exposed than in a raw engine like llama.cpp.

Best for: Anyone who wants API-first local inference with the least friction between install and first response.

3. LM Studio

LM Studio trades the command line for a polished graphical interface, with model discovery plus controls for chat history and parameters. It is proprietary but free for personal and work use, which makes it a common pick for developers who prefer a desktop app.

Key features:

• Graphical model management: Discover, download, and tune models without touching a terminal.

• Standard server on port 1234: It serves OpenAI-compatible endpoints that any OpenAI client can call.

Pros:

• Friendly interface with llama.cpp and MLX backends under the hood.

• Free for both personal and commercial use.

Cons:

• The application is proprietary, so you cannot inspect or fork the code.

• A desktop-first design is less natural for headless server deployments.

Best for: Developers who want a visual way to browse models and run them, plus an API server when they need one.

4. llama.cpp

Underneath many of these tools sits llama.cpp, the inference engine that does the actual work. Its llama-server binary exposes an OpenAI-compatible REST API directly, so you can run it on its own when you want to control every layer.

Key features:

• Direct server binary: llama-server gives you an OpenAI-compatible REST API with no wrapper in the way.

• Broad GPU backends: Its build documentation covers Metal, CUDA, SYCL, and Vulkan.

Pros:

• Maximum control over quantization, context, and serving parameters.

• Runs almost anywhere, from Apple Silicon to CPU-only servers.

Cons:

• You handle build flags and configuration yourself, so the learning curve is steeper.

• No graphical layer, which slows casual experimentation.

Best for: Engineers who want maximum control or are building their own tools on top of the underlying engine.

5. Jan

Jan is built for teams that treat privacy as non-negotiable. It runs fully offline as an open-source, Apache 2.0 licensed chat application. It pulls local models and speaks the same OpenAI schema as the rest of this list, with optional cloud bridges when you want a hybrid workflow.

Key features:

• Offline-first with an open license: Apache 2.0 licensing and air-gapped operation suit verifiable-privacy requirements.

• API on localhost:1337 plus MCP: It exposes an OpenAI-compatible API and supports the Model Context Protocol (MCP) for tool calling.

Pros:

• Genuinely offline operation for air-gapped environments.

• Permissive Apache 2.0 license you can audit and self-host.

Cons:

• A younger project with a smaller community than Ollama.

• Optional cloud bridges reintroduce data-egress questions if you turn them on.

Best for: Teams that need verifiable privacy and air-gapped operation as a hard requirement.

6. GPT4All

GPT4All from Nomic AI targets everyday desktops and runs models with no GPU required. Its documentation walks through private desktop use with no-configuration setup, which lowers the bar for a first local model.

Key features:

• No-GPU desktop use: It runs on ordinary hardware with a no-configuration setup path.

• LocalDocs for your files: Point it at a folder of documents and chat with them fully offline.

Pros:

• Very low barrier for a first local model on modest hardware.

• LocalDocs adds private document search without cloud services.

Cons:

• Not aimed at server or agentic deployments the way the runtimes above are.

• Consumer-desktop focus limits throughput for heavier workloads.

Best for: People getting started who want a straightforward desktop app and offline document chat.

The best local LLM models to run in 2026

Model families change quickly, so treat any list as a snapshot and check current releases. As of publication, strong candidates for local coding and agentic workflows include Qwen coding models, Mistral Devstral, small Phi models for constrained machines, and current Gemma-family models sized to your hardware. Public coding benchmarks help you compare performance, but the hardest reasoning tasks and extended contexts still need evaluation against your own work before you trust them in production.

General-purpose model families

These families span laptop through server tiers, so you can almost always find a size that fits your hardware without dropping to a toy model.

• Qwen: Dense and mixture-of-experts (MoE) models across many sizes, with Apache 2.0 licensing on the Qwen3 family. Look here first for local models across laptop, workstation, and server tiers from one lineage.

• DeepSeek: Large MoE models alongside distilled dense variants that run on smaller hardware, with details in the DeepSeek R1 model card. On a single-GPU machine, the distilled variants are the ones to try.

• Gemma: Google's open Gemma 3 family covers 1B, 4B, 12B, and 27B sizes, a clean set of steps from laptop to workstation.

• Mistral and Phi: Mistral publishes official announcements including Mistral 3, and Microsoft's Phi-4-mini targets low-resource machines when you need something that runs on modest hardware.

Most of these lineages also ship coding-tuned variants, and those are where to look next if the model's main job is writing software.

Models tuned for coding

For fully local coding, prioritize models that fit your hardware, handle tool calls reliably, and hold up on your own repositories rather than someone else's leaderboard. Qwen Coder variants are strong candidates for 24 GB-class systems, and Mistral's Devstral release targets agentic software engineering tasks specifically. One caveat outweighs almost everything else for agentic work, and it is tool calling. Reliability varies more by model family than by parameter count. Tool-calling reliability varies by model and runtime, so test it directly rather than relying only on model size or general coding benchmarks.

Matching a model to your hardware tier

Read available memory, model size, and quantization level together, because any one of them on its own will mislead you. The mapping below gives you a practical starting point for the tier you already have.

Whichever tier you land on, the model itself is only half of the setup. OpenHands can use a local model as the inference backend for a coding agent workflow through its LiteLLM-compatible model configuration. That means the model you fit onto your hardware can power agent workflows against real repositories, with output still subject to testing and human review.

How to set up your first local LLM

Ollama gets you to a working local model with the least setup, so start there, and a single command is all it takes to install it.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# Windows: download the installer from https://ollama.com/download

Pull and run the Llama 3.2 model next, using an explicit tag such as llama3.2:1b or llama3.2:3b to match a size to your hardware. On most laptops you will want to start small and move up gradually if you have more memory to work with.

ollama pull llama3.2:3b
ollama run llama3.2:3b

Ollama exposes an OpenAI-compatible API automatically, so you point any OpenAI client at it and keep the rest of your code unchanged. Changing base_url is the whole trick, and from there anything that speaks the OpenAI schema runs against your local model instead of a remote one.

from openai import OpenAI
client = OpenAI(
base_url='http://localhost:11434/v1',
api_key='ollama',
)
response = client.chat.completions.create(
model='llama3.2:3b',
messages=[{'role': 'user', 'content': 'Hello!'}]
)

A reply back from that script means your machine is serving a working OpenAI-compatible model. Everything else in this guide builds on that endpoint, starting with handing it to a coding agent.

How to run a local LLM with an AI coding agent

The integration pattern stays the same across local LLM tools, and that consistency is what makes running an agent locally realistic. Your local server exposes an OpenAI-compatible endpoint, and your coding agent points at the local URL instead of api.openai.com, so it behaves the same whether the model runs on your laptop or in a data center. You will point at one of these endpoints depending on the runtime you chose.

• Ollama: http://localhost:11434/v1, API key 'ollama', through its OpenAI-compatible endpoint.

• LM Studio: http://localhost:1234/v1, through its OpenAI-compatible server on port 1234.

• llama.cpp: localhost:8080, through the llama.cpp server binary.

LiteLLM Proxy sits in front of all of them as a universal translation layer. It is fully OpenAI-compatible and works with any tool that calls OpenAI, so you change the connection settings and the model name and leave the tool itself alone.

model_list:
- model_name: "deepseek-r1"
litellm_params:
model: "ollama_chat/deepseek-r1"
api_base: "http://localhost:11434"

Plan around two limits before you commit an agent to a local model. Tool calling varies by model, so test it directly rather than trusting the spec sheet, and confirm the effective context window in your runtime rather than assuming the advertised context is what your workflow gets.

Running local and cloud models on one platform with OpenHands

OpenHands can use local or hosted models as inference backends for coding-agent workflows. That lets teams route sensitive work to a local or private model and use hosted APIs for tasks where they need top-tier reasoning, speed, or convenience.

For example, OpenHands’ AMD Ryzen AI setup shows the agent running against a local Qwen coding model through Lemonade Server. The model server handles local inference, while OpenHands provides the coding-agent workflow on top.

The residency line matters. A workflow that uses a local model can keep inference inside the environment you control. A workflow that calls a hosted model provider may send prompts and code context outside that environment. Teams should evaluate the full data path, including model inference, tools, logs, telemetry, and any integrations the agent can call.

For teams under governance, OpenHands Enterprise adds the control layer for running agents across projects and teams, including access controls, auditability, usage visibility, budgets, and sandboxed runtimes. That lets platform and security teams define how agents run without forcing every developer to manage local infrastructure by hand.

Choosing a local LLM setup that fits how you build

Start with the constraint that actually binds you, because it settles most of the rest of the decision. If you chase every option at once, you end up with hardware you do not need and a workflow nobody on your team uses.

If compliance or data sovereignty is your driver, the question is already answered, and you run local or self-hosted. When cost is your driver, run the volume math first, since light or unpredictable usage often justifies a cloud API while steady, sensitive, high-volume workloads tilt toward self-hosting.

If latency or rate limits are your driver, especially for agentic workflows firing many calls per task, local inference removes both the round trip and the throttling in one move. For most teams the answer is a hybrid approach that keeps local models on sensitive work and reaches for cloud APIs when a task still needs top-tier reasoning.

The best local setup isn't the biggest model you can fit into memory, it's the one that matches your operational constraints. Start with the workflow you want to run, then choose the model, runtime, and deployment approach that best supports it.

If you're ready to put that workflow into practice, OpenHands can use local or hosted models as the inference backend for coding-agent workflows, letting you start locally and scale to shared, governed deployments without changing platforms.

Frequently asked questions about local LLMs

Are local LLMs as good as ChatGPT or Claude?

For plenty of production tasks, local models are worth taking seriously. Structured extraction, summarization, and standard code generation often land close enough, and proprietary models still lead on the hardest reasoning and extended contexts. Many teams run both side by side, routing each task to the right model with in-conversation model choice.

Can I run a local LLM without a GPU?

Yes, CPU-only inference with GGUF-quantized models works but runs slower, so small models are the practical choice. Memory bandwidth is the usual bottleneck, and faster memory often helps more than extra capacity once the model fits. From there, an agent running in the outer loop can put even a modest local model to work.

What is the best local LLM for coding?

There is no universal best model, only the right pick for your hardware and repositories. On a single 24 GB GPU, quantized 24B–32B coding models are the category to evaluate, and a laptop calls for something smaller tuned for latency. When agentic coding leans on tool calls, weigh tool-calling accuracy over benchmark scores. Once you know which model fits, you can compare deployment options from there.

Is running a local LLM actually free?

Free model weights still leave you paying for engineering, infrastructure, electricity, and maintenance. Cloud APIs are often easier to justify at light or unpredictable usage, and local wins when privacy, latency, or sustained volume dominates. For central control over spend, the enterprise control layer adds budgets and cost attribution across projects.

About OpenHands

OpenHands is the open-source platform for building and running AI coding agents, with the interface, automations, and control layer needed to go from a single local agent to a system running across an entire organization. The mission is to make agent-based software development accessible, transparent, and controllable by default. That starts in the open. The core framework is open source, giving developers and platform teams full visibility into how agents execute work and interact with their systems. The project has over 80,000 GitHub stars, over 9 million downloads, and contributions from hundreds of developers. OpenHands is used by engineers at large enterprises and fast-growing startups to build, run, and scale AI coding agents across real software engineering workflows. The long-term vision is to become the full stack AI coding agent platform for software engineering. Not just helping developers write code, but running meaningful parts of the software lifecycle.