Your compliance team just blocked the cloud AI tool half your developers were using, and the CISO wants to know where every prompt containing source code goes.

Self-hosting an AI model gives your organization direct control over where model inference runs, which model weights are deployed, and how the inference service is operated. When the surrounding architecture is also configured to remain private, prompts and outputs can stay inside infrastructure your organization controls.

That control comes with a tradeoff: your team now owns the GPUs, inference runtime, scaling, monitoring, patching, access controls, and model lifecycle that a hosted provider would otherwise manage.

This guide explains what self-hosting actually involves, from hardware and inference runtimes to security and deployment, and where the model layer fits once coding agents start acting on its output.

What does it mean to self-host an AI model?

Self-hosting an AI model means running model inference on infrastructure your organization controls rather than sending requests to a third-party hosted model API.

Typically, teams deploy open-weight model checkpoints on local workstations, private servers, or cloud infrastructure they control, then serve the model through an inference runtime such as vLLM, Ollama, or llama.cpp.

Applications can call that self-hosted endpoint much like they would call a hosted model API. Several popular inference runtimes expose OpenAI-compatible APIs, which can reduce the application changes required when switching inference backends.

Self-hosting is not the same as running a model locally. A developer running a model on a laptop is one form of self-hosting, but an enterprise deployment might serve the same model from a GPU cluster inside a private VPC or data center.

The important distinction is control. With a hosted API, model inference runs on infrastructure operated by another provider. With self-hosting, your organization takes responsibility for the infrastructure and model-serving layer.

Why self-host AI models?

Teams generally self-host for four reasons: greater control over data and infrastructure, economics at sustained utilization, model customization, and control over the model lifecycle.

Data control and regulatory requirements

For organizations handling source code, customer data, health information, financial data, or other sensitive workloads, the location and governance of model inference can become part of the security and compliance review.

Hosted APIs introduce another service provider into the data path. Depending on the workload and jurisdiction, teams may need to evaluate where prompts are processed, how long data is retained, who can access it, what subprocessors are involved, and what contractual or technical controls apply.

Self-hosted inference gives organizations more direct control over that layer. Teams can determine where inference runs, how requests are logged, which model checkpoint is deployed, and how network access is restricted.

Self-hosting does not itself establish compliance or data sovereignty. The complete data boundary still depends on the applications calling the model, connected tools, agent runtimes, telemetry, logging, storage, backups, and other infrastructure around it.

Predictable costs at sustained utilization

Self-hosting changes the economics of inference.

Hosted APIs generally turn inference into a variable expense based on usage. Self-hosting shifts more of that cost toward infrastructure and operations. Your organization pays for accelerator capacity whether the GPUs are busy or idle.

At sustained utilization, that can make self-hosting economically attractive. At low or bursty utilization, hosted inference may still be cheaper because the provider absorbs idle capacity.

The comparison also depends heavily on what you’re replacing. Competing with a premium frontier-model API produces different economics from competing with a low-cost hosted open-weight model.

Do not compare token prices alone. A realistic total-cost calculation should include:

• GPU or accelerator costs

• idle capacity

• storage and networking

• infrastructure engineering

• monitoring and observability

• upgrades and patching

• capacity planning

• incident response

For many teams, engineering time is one of the easiest costs to underestimate.

Fine-tuning and customization

Open-weight models give teams more control over how a model is modified and deployed, subject to the license of the specific checkpoint.

Depending on the model, teams may use full fine-tuning, LoRA, or other parameter-efficient methods to adapt it to a particular workload.

Self-hosting is especially relevant when an organization wants the resulting weights and inference environment to remain inside infrastructure it controls. But fine-tuning does not inherently require self-hosting; some model providers also offer managed fine-tuning services.

Always evaluate the license of the specific model and checkpoint before modifying or deploying it commercially. “Open weight” does not imply that every model carries the same usage, redistribution, or derivative-work rights.

Control over the model lifecycle

Hosted model providers control their own rate limits, pricing, availability, and deprecation schedules.

Self-hosting moves more of those decisions to your organization. Teams decide when to upgrade a model, when to test a new checkpoint, how much inference capacity to provision, and when an older model should be retired.

That doesn’t eliminate change. It changes who owns it. Your team now has to evaluate new checkpoints, test upgrades, patch the serving infrastructure, manage capacity, and maintain compatibility with the applications using the endpoint.

Self-hosted AI vs. cloud APIs: how the tradeoffs compare

The decision is less about whether one approach is universally better and more about which responsibilities your organization wants to own.

Self-hosting also doesn’t give you access to every model. Proprietary frontier models generally remain available only through their providers’ services.

The tradeoff is straightforward: self-hosting gives you more infrastructure control by giving you more infrastructure responsibility.

Hardware requirements for self-hosting AI models

Once you’ve decided to self-host, hardware determines which models you can run, how much context they can process, and how many requests you can serve concurrently.

The same model that works for one developer on a workstation may require multiple accelerators when it becomes a shared production endpoint.

How much memory do you need?

A useful starting point for model weights is:

model parameters × bytes per parameter ≈ memory required for weights

As a rough theoretical approximation before quantization metadata and other overhead:

• FP16: 2 bytes per parameter

• INT8: 1 byte per parameter

• INT4: 0.5 bytes per parameter

A 7-billion-parameter model at FP16 therefore needs roughly 14 GB just for the model weights. Quantizing the same model reduces that requirement substantially.

But model weights are only the starting point.

A more realistic capacity model is:

model weights + KV cache + runtime overhead + concurrency headroom = required accelerator memory

The KV cache stores information used during generation and grows with context length and concurrent requests. That’s why a model that comfortably fits during a single-user test can run out of memory once it becomes a shared endpoint handling long-context requests.

Always size against the actual model, context window, quantization, runtime, and expected concurrency rather than parameter count alone.

Quantization: fitting larger models into less memory

Quantization represents model weights at lower numerical precision, reducing the amount of memory required to load and serve the model. Four-bit quantization is a common starting point for local deployments because it can substantially reduce memory requirements while preserving useful quality for many workloads.

Lower precision is not free, however. The quality impact varies by model, quantization method, and workload. A configuration that performs well for conversational tasks may behave differently on code generation, mathematical reasoning, or tool use.

Higher-precision formats may therefore be preferable when small quality regressions matter and hardware allows them.

Do not choose a quantization level from a generic benchmark alone. Evaluate the exact checkpoint and serving configuration against representative tasks from your workload.

How to choose a model to self-host

Model selection should start with the workload, not the leaderboard. A model that performs well on a general reasoning benchmark may not be the right choice for a coding agent, retrieval workflow, support assistant, or high-throughput classification service.

Evaluate at least these factors:

• Task performance: Test the model against examples that resemble the work it will actually perform.

• Memory requirements: Consider weights, context length, concurrency, and serving overhead together.

• Latency and throughput: A model that fits on your hardware may still be too slow at the concurrency your users require.

• Context requirements: Longer context windows increase memory pressure and can materially change serving economics.

• Quantization support: Check which quantization formats are supported by both the model and your intended runtime.

• Tool use and structured output: Agent workloads often depend on reliable tool calling and structured responses, not just text-generation quality.

• Runtime support: Verify that the checkpoint works well with your intended serving stack and hardware.

• License: Check commercial-use rights, redistribution terms, derivative-work requirements, and any model-specific restrictions.

For coding workloads, open-weight model families from organizations such as DeepSeek, Qwen, Mistral, OpenAI, and Moonshot AI provide different tradeoffs in size, performance, licensing, and hardware requirements.

Treat published benchmark results as a shortlist rather than a buying decision. Agent benchmarks are particularly sensitive to the scaffold, prompts, tools, inference settings, and evaluation harness around the model.

Benchmarks build the shortlist. Your workload should pick the model.

Tools and runtimes for serving self-hosted models

Once you’ve selected a model, you need an inference runtime to load the weights and expose them to applications.

The right runtime depends on your hardware, concurrency requirements, and how much infrastructure you want to operate.

vLLM

vLLM is commonly used when teams need to turn GPU infrastructure into a shared inference service. It supports capabilities such as continuous batching, caching, distributed inference, and OpenAI-compatible APIs. That makes it a useful option for team-shared or production endpoints where throughput and concurrency matter.

Ollama

Ollama emphasizes local setup and model management. For many developers, it is one of the simplest ways to get an open-weight model running on a workstation and expose it through a local API. It is particularly useful for development and experimentation, though teams should evaluate authentication, scaling, observability, and infrastructure requirements separately before using any local-first runtime as a shared production service.

llama.cpp

llama.cpp is particularly useful for quantized models running across CPUs, Apple silicon, GPUs, and other mixed hardware environments. Its support for GGUF models and lower-resource hardware makes it useful for local development, edge environments, and cases where dedicated data-center GPUs are unavailable. CPU inference is generally slower and becomes harder to scale as model size and concurrency increase, so test throughput against the actual workload.

LocalAI

LocalAI provides a common API layer across multiple self-hosted model types and modalities. It can be useful when teams want one interface over several self-hosted AI capabilities rather than operating a separate API for each model type.

The important distinction is that all of these tools solve the inference layer. They load a model, process requests, and return output. They do not decide what an AI agent is allowed to do with a repository, which tools it can call, where generated code executes, how changes are validated, or when a human needs to approve an action.

That becomes important when the client calling your self-hosted model is an AI agent rather than a chat interface.

How to self-host an AI model: step-by-step

The exact deployment will vary by model, runtime, and infrastructure, but the process generally follows the same path: choose the model, make sure it fits your hardware, select a serving runtime, deploy an internal endpoint, and monitor it under real traffic.

1. Choose the model and precision

Start with the workload rather than the biggest model you can fit. Select a model based on task performance, license, tool-use requirements, and runtime support. Then determine whether you can run it at its native precision or need quantization to fit your available hardware.

Use model weights as the baseline for memory requirements, then add capacity for KV cache, concurrent requests, and runtime overhead:

model weights + KV cache + runtime overhead + concurrency headroom = required accelerator memory

Start smaller than you think you need. A model that runs reliably at useful latency is often more valuable than a larger model that consumes nearly all available memory before real traffic arrives.

For coding agents in particular, evaluate more than code-completion benchmarks. Test the model’s ability to follow instructions, use tools, work across files, recover from failed attempts, and complete representative tasks from your own repositories.

2. Pick a runtime

Match the runtime to the deployment.

For local development, Ollama or llama.cpp can minimize setup overhead. They make it relatively easy to download a model, run it on a workstation, and expose a local API.

For a shared GPU endpoint with higher concurrency, a production-oriented runtime such as vLLM may be more appropriate. It supports capabilities such as continuous batching, caching, distributed inference, and OpenAI-compatible APIs.

The runtime should support your chosen model, hardware, quantization format, API requirements, and monitoring stack.

3. Choose where inference runs

A self-hosted model can run on:

• a developer workstation

• an internal GPU server

• a private cloud instance

• a Kubernetes cluster

• an on-premises data center

• other infrastructure controlled by your organization

Where it runs should follow your performance, cost, availability, and data-boundary requirements.

“Self-hosted” does not necessarily mean “on-premises.” A model running in your organization’s private cloud environment can still be self-hosted if your team operates the inference service and controls the surrounding infrastructure.

4. Deploy, connect, and monitor the endpoint

Getting the model running is often the easiest part. For example, vLLM can expose a supported model through an OpenAI-compatible API with a command like:

vllm serve <model-id> \
  --served-model-name coding-model \
  --api-key $VLLM_API_KEY

Replace <model-id> with the model checkpoint you want to serve. Check the model’s current vLLM compatibility and hardware requirements before deploying it.

The --served-model-name flag gives clients a stable name for the model rather than requiring them to use the underlying model path. That makes it easier to change the backing checkpoint later without changing every client configuration.

Applications and agent clients that support the OpenAI API can then connect to the endpoint using its base URL and configured credentials.

For OpenHands, that means the self-hosted model becomes the inference backend for the software engineering agent rather than the agent itself. The model generates the next response or action; OpenHands provides the surrounding agent workflow that can interact with repositories, tools, commands, tests, and other parts of the development environment.

In a production environment, do not treat the runtime’s API key as the complete security boundary. Put the inference server behind an internal gateway or reverse proxy rather than exposing the serving process directly to every client. The gateway can provide authentication, routing, rate limiting, and other controls appropriate to your environment.

Once real traffic arrives, monitor more than whether the endpoint is up. Track:

• request volume and error rates

• request latency and time to first token

• token throughput

• queue depth

• accelerator utilization

• accelerator memory

• cache behavior

Queue depth is particularly useful under shared load. If requests consistently arrive faster than the hardware can process them, the deployment has exceeded its useful concurrency. You may need additional capacity, a smaller model, a different quantization, or tighter request limits.

Also monitor model-level outcomes. Infrastructure metrics can tell you whether the endpoint is healthy, but they cannot tell you whether a new checkpoint or quantization level made the model worse at the job it is supposed to perform.

5. Test the deployment before expanding access

A successful API response only proves that the model is running. Before treating the endpoint as production infrastructure, test the complete path that applications will use.

Verify that:

• authentication works as intended

• unauthorized clients cannot reach the endpoint

• requests and responses follow your logging and retention policies

• the model performs adequately under concurrent load

• failures and timeouts are handled cleanly

• monitoring captures the metrics your team needs

• the selected model and quantization perform well on representative tasks

For coding-agent workloads, run a small evaluation set drawn from real software engineering tasks rather than testing only isolated prompts. A model may look strong in a chat interface but struggle when it has to repeatedly select tools, interpret their output, modify multiple files, run tests, and recover from errors.

Once the endpoint is reliable, the next question is security. Self-hosting moves responsibility for network isolation, authentication, encryption, patching, model artifacts, and logging from the model provider to your team.

Security best practices for self-hosted AI

Self-hosting moves security responsibility from the API provider to your organization.

An inference endpoint should therefore be treated like any other production service handling sensitive application data.

Isolate the inference service

Keep inference servers on private networks whenever possible rather than exposing serving ports directly to the public internet. Use network segmentation, private subnets, gateways, or other controls appropriate to your infrastructure so only authorized applications can reach the service.

Authenticate at the service boundary

Do not assume the inference runtime provides sufficient authentication or authorization for your environment. Put shared endpoints behind an authenticated gateway or service layer and define which users, applications, and workloads can call which models.

Encrypt traffic

Protect requests between applications, gateways, inference services, and logging infrastructure using encrypted transport appropriate to your architecture. For sensitive environments, evaluate whether service-to-service authentication is also required.

Patch inference infrastructure

Inference servers are network services that parse complex requests, load model artifacts, and often interact with GPU and distributed-computing frameworks. Track security advisories for the runtime and its dependencies, test updates, and establish a defined patching process.

Verify the model supply chain

Model files are software artifacts and should be treated accordingly. Verify where checkpoints came from, pin versions or revisions where appropriate, prefer safer serialization formats such as SafeTensors when available, and avoid loading untrusted executable model artifacts into production infrastructure.

Control logging and retention

Self-hosting gives your organization control over inference logs, but that creates another security decision. Determine whether prompts, outputs, identifiers, or metadata should be logged at all, who can access those records, how long they should be retained, and whether sensitive source code or user data could appear in them.

Together, these controls protect the inference layer. They do not answer a separate set of questions: Who can point an autonomous agent at the model? What systems can that agent access? Where does its code execute? What did it change?

Those belong to the agent layer.

Self-hosting AI models for coding agents

A hardened self-hosted endpoint answers one important question: Where does model inference happen?

It does not answer what an AI agent can do once it receives the model’s output.

A coding agent may read repositories, modify files, execute shell commands, run tests, access external tools, interact with source control, and open pull requests. That introduces an execution layer on top of inference.

The architecture now looks something like:

Model weights → inference runtime → private endpoint → agent → execution environment → tools and repositories → validation and review

Each layer has a different job.

The inference runtime serves the model. The agent turns model output into actions. The execution environment determines where those actions run. Tools and integrations determine what the agent can reach. Validation determines whether the resulting work is acceptable.

Self-hosting a model gives you control over inference. It does not, by itself, give you control over what an AI agent does with that model.

Where OpenHands fits

OpenHands is an open-source platform for building and running software engineering agents.

Teams can connect the OpenHands agent to supported self-hosted model endpoints and use those models as the inference backend for workflows that read repositories, modify code, run commands and tests, and produce reviewable changes.

This separates two architectural decisions:

Model layer: Which model should perform inference, and where should it run?

Agent layer: What should the model be allowed to do, where should its actions execute, and how should the resulting workflow be observed and governed?

This separation lets teams choose the model and inference infrastructure that fit their requirements without tying the agent workflow to a single model provider. Teams can use supported self-hosted models for private inference, hosted models when frontier performance matters, or change the model layer as their requirements evolve.

For organizations scaling these workflows, OpenHands Enterprise adds organization-level controls around the agent layer. The Agent Control Plane provides centralized policies, sandboxed execution, workflow visibility, records of agent activity, cost attribution, and controls for running repeatable automations across repositories and teams.

Those controls complement the security of the inference endpoint rather than replacing it.

The inference layer governs access to the model. The agent layer governs how model-powered workflows interact with code, tools, repositories, and execution environments.

Pairing a self-hosted OpenHands deployment with private model inference can keep more of the agent workflow inside infrastructure the organization controls. Whether the full data path remains private still depends on every connected component, including source-control systems, tools, MCP servers, telemetry, logging, storage, and other integrations.

Model choice should therefore be part of the security architecture as well as the performance evaluation.

Choosing the right self-hosting architecture

There is no single self-hosting stack that makes sense for every team. Start with the constraint that is actually driving the decision.

If you’re experimenting locally

Start with a runtime such as Ollama or llama.cpp and a model that comfortably fits your available memory.

Optimize for simplicity before throughput. The goal is to learn whether the model works for the workload before building infrastructure around it.

If you’re serving a team

Move inference onto shared infrastructure, add authentication and monitoring, and evaluate a production-oriented runtime such as vLLM.

Capacity planning now matters because context length and concurrent requests can make a configuration that worked for one developer behave very differently under shared load.

If data containment is the primary requirement

Map the complete data path before choosing components.

Self-hosting the model is only one part. The applications, agent runtime, tools, logging, storage, telemetry, and integrations also need to fit within the required boundary.

If cost is the primary requirement

Compare total cost of ownership against hosted inference using your actual utilization.

Include accelerator costs, idle capacity, infrastructure engineering, monitoring, patching, upgrades, and incident response rather than comparing token prices alone.

At lower or bursty utilization, hosted inference may still be more economical.

If coding agents will use the model

Treat the inference service and agent platform as separate layers.

Secure the model endpoint, then define how agents can execute code, access repositories and tools, validate changes, and record workflow activity.

OpenHands provides the agent layer while preserving flexibility in the model and inference backend, allowing teams to start with a model that fits today’s workload and change the inference backend as requirements evolve.

Frequently asked questions about self-hosting AI models

Is self-hosted AI the same as local AI?

No. Local AI typically refers to models running directly on a user’s device or workstation. Self-hosted AI is broader: the model can run on a laptop, private server, cloud infrastructure your organization controls, or an on-premises cluster. The defining characteristic is who operates the inference infrastructure, not whether the hardware is physically local to the user.

Is self-hosting an AI model cheaper than using an API?

It can be, but utilization matters. Self-hosting becomes more economically attractive when workloads keep expensive hardware consistently busy. At lower or bursty usage, hosted APIs can be cheaper because you only pay for inference when you use it.

Compare total cost of ownership, including hardware or cloud accelerators, idle capacity, engineering time, monitoring, upgrades, patching, and incident response.

Can you self-host AI models without a GPU?

Yes. Quantized models can run on CPUs and other hardware supported by runtimes such as llama.cpp. CPU inference is generally slower and becomes harder to scale as model size, context length, and concurrency increase.

For local experimentation or smaller workloads, that may be perfectly acceptable. Shared production workloads should be benchmarked against their actual latency and throughput requirements.

What's the easiest way to self-host an AI model?

For many developers, Ollama is one of the simplest ways to start locally because it handles much of the model download, management, and serving setup.

llama.cpp is another common choice when hardware flexibility and quantization are priorities.

As usage expands beyond a single developer, evaluate the serving stack based on concurrency, authentication, monitoring, accelerator support, and operational requirements rather than ease of installation alone.

Is self-hosting AI models secure enough for regulated industries?

Self-hosting can give organizations more direct control over where inference runs, how network access is restricted, and how logs and model artifacts are managed.

It does not automatically make a deployment secure, compliant, or sovereign. Organizations still need to secure the inference service and evaluate the complete data path, including applications, agent runtimes, integrations, telemetry, storage, logging, and model supply chain. The controls required depend on the workload, architecture, jurisdiction, and applicable regulatory requirements.

Does self-hosting keep all AI data inside my environment?

Not necessarily. Self-hosting can keep model inference inside infrastructure you control, but the full workflow may still communicate with external services.

For example, an AI agent might use a self-hosted model while connecting to externally hosted source control, MCP servers, observability systems, or other APIs.

If data containment is a requirement, map every component that can receive prompts, code, model outputs, logs, or other workflow data.

Self-host the model. Control the workflow.

Self-hosting gives teams direct control over one of the most important layers of an AI system: inference. But the model is only one layer.

Once AI agents start reading repositories, executing commands, modifying code, and interacting with engineering systems, teams also need to think about execution environments, permissions, validation, observability, and governance.

That is the distinction to keep in mind when designing a private AI stack: Self-hosting the model controls where inference happens. Controlling the agent determines what happens next.

OpenHands lets teams run software engineering agents against supported self-hosted models while keeping model choice separate from the agent workflow. Developers can start locally, and organizations that need centralized controls can deploy OpenHands Enterprise inside infrastructure they manage.