When someone asks about running an on-premise LLM, the phrase usually holds two different problems packed together, and confusing them is where most hardware budgets and data-residency assumptions go wrong. Keeping the orchestration layer on your own server and keeping model inference on your own server are separate engineering decisions. They have different hardware floors, different network boundaries, and different consequences when a plan turns out to be wrong.
Short answer: You can run the orchestration layer, your stored data, your workflows, and your conversation history entirely on your own server from as little as 2 vCPU, 4 GB RAM, and 20 GB SSD. If you also want model inference to stay on-premise, you need hardware sized to the specific model weights you plan to run, which is a separate and substantially larger decision. Most deployments start in hybrid mode: the runtime and all stored data remain on your infrastructure, and inference requests route to external frontier model gateways under those providers' commercial zero-retention API policies.
How it works, in plain terms
| Aspect | Hybrid mode | Fully local mode |
|---|---|---|
| Minimum server spec | 2 vCPU, 4 GB RAM, 20 GB SSD | Sized to the model weights, typically dedicated CPU or GPU compute |
| Runtime and stored data on your server | Yes | Yes |
| Inference on your server | No | Yes |
| Reasoning depends on an external API | Yes | No |
| Prompts cross the network | Yes, on each inference request | No |
| Per-seat licensing | None | None |
Hybrid is where nearly every team starts, and the split is narrower than the name suggests. The runtime that coordinates agents, the task tracking, the documents with their version history, the conversation history, the memory used for retrieval, and the append-only event log all live on your server. Credentials stay in the local secret store and are resolved at execution time under a strict allowlist; they are never placed in the text of a prompt. What leaves is the context payload for one reasoning request, sent to the model gateway you configured.
That is where the boundary actually sits: at the inference call, not at the data store. A record you save stays saved on your hardware. The excerpt an agent needs in order to answer a question is the thing that travels out, and only for as long as that request takes.
Warning: In hybrid mode, the context payload sent to a frontier model gateway for each inference call traverses the network to that provider. An on-premise deployment does not mean zero external data transit unless it is explicitly configured with exclusively local model weights and no outbound gateway connections. Telling stakeholders or auditors otherwise before verifying the configuration is a compliance and trust risk.
Choosing between those two columns is one decision inside a larger architecture. The runtime, agents, roles, tasks, channels, workflows, and policies that coordinate the work around that inference layer are covered on the self-hosted AI agents page; this article stays on the inference and hardware question.
A worked example
A twelve person operations team wants help with email triage and the weekly reporting pack. Nobody is willing to pay per seat for that, and nobody can yet justify a GPU server. They deploy a single hybrid node on a VPS at 2 vCPU, 4 GB RAM, and 20 GB SSD.
- The orchestration runtime, task tracking, document storage with version history, retrieval memory, and the append-only event log run on that VPS.
- Several agents share the node. Each runs in its own isolated process, with its own declared tool grants and its own capability limits. One handles email triage, another drafts reports, and each is assigned the model that suits its role.
- API keys and mailbox passwords sit in the local secret store and are resolved at execution time, so they never appear inside a prompt.
- Before an agent sends an external email, publishes content, deletes a record, or runs a privileged command, a human-in-the-loop gate stops the action and waits for a person to confirm it.
- The inference egress event is the context payload sent to the model gateway for each reasoning call. Channel traffic, such as outbound email or Telegram messages, is a separate egress path that traverses those external networks. The runtime, the stored documents, and the conversation history stay on the server.
- Licensing does not change with headcount. The same node serves ten people or forty.
What the team owns at the end is one Linux server to patch, one place where its records live, and a countable network path for reasoning. That last part is the one worth putting in writing for anyone who asks about data residency.
Steps to apply it
- Decide the inference mode: hybrid or fully local. You now have the single number that sets everything else, because this choice determines your hardware floor.
- Provision a Linux server on Ubuntu or Debian that meets the minimum for that mode. For hybrid, a VPS at 2 vCPU, 4 GB RAM, and 20 GB SSD is the floor. For fully local, size dedicated CPU or GPU compute against the model you intend to run, and identify that model before you order anything.
- Assign operational ownership. Your organization, or the managed service provider acting for it, is accountable for host security, OS updates, monitoring, backups, and recovery. You now have a named person for each of those five tasks rather than an assumption that someone handles them.
- Follow the prerequisites and installation runbook at /docs/install to bring the node up. You now have a running node with its base services in place.
- Load your external model gateway credentials into the secret store, or configure your local model runner if you went fully local. You now have working inference for at least one agent.
- Define each agent with a declared role, a tool grant list limited to what that role needs, and a model assignment. You now have bounded autonomy rather than open-ended access.
- Configure human-in-the-loop policies for every action that is irreversible or externally visible: outbound email, publishing, record deletion, privileged commands, and monetary spend. You now have a gate that holds those actions for confirmation.
- Connect the channels your team will actually use, whether that is the browser chat panel, SMTP and IMAP email, Telegram, or voice, and verify that each one routes conversations to the correct agent. You now have a node your team can reach.
Common mistakes
Three planning errors account for most failed on-premise LLM projects, and each one is cheap to catch before hardware is ordered.
The first is treating the hybrid hardware floor as the budget for local inference.
Warning: 2 vCPU, 4 GB RAM, and 20 GB SSD is the minimum for orchestration with remote inference. Running model weights on the same machine requires hardware sized to those weights, which is a separate and larger calculation. Ordering a VPS at the hybrid minimum and then expecting to run inference locally is the most common single cause of a stalled project.
The second is assuming that on-premise means nothing ever crosses the network.
Warning: Data residency claims need to match the configuration that is actually deployed. Verify which model gateways are reachable and which agents use them before making a promise to a customer, an auditor, or your own board.
The third is reading an installation as a compliance certificate.
Warning: Installing the software does not by itself produce HIPAA, GDPR, or SOC 2 compliance. Those depend on the infrastructure configuration, access policies, data retention practices, and administrative procedures around the node.
Frequently asked questions
Does running an LLM on-premise mean my data never leaves my server?
Not in hybrid mode. Your stored documents, conversation history, retrieval memory, and event log stay on your server, but the context payload sent to a frontier model gateway for each inference call crosses the network to that provider, governed by its commercial zero-retention policy. Fully local mode, running exclusively local model weights with no external gateway, keeps inference on your server as well. The security architecture page sets out where each boundary falls in more detail.
What is the minimum hardware to start?
For hybrid mode, 2 vCPU, 4 GB RAM, and 20 GB SSD on a Linux VPS running Ubuntu or Debian is the documented minimum. For fully local inference, the requirement is sized to the model weights you intend to run and usually means dedicated CPU or GPU compute.
Can I run some agents on a local model and others on an external API at the same time?
Yes. Different agents on the same node can be configured with different models, so a lightweight task can route to a local model runner while a task that needs heavier reasoning routes to a frontier gateway.
Does on-premise AI eliminate all costs?
It eliminates per-seat subscription fees, because licensing does not scale with headcount. Infrastructure hosting, server maintenance, and per-token charges from any frontier model gateway you use remain. Fully local inference removes gateway token costs but replaces them with hardware investment sized to the chosen model.
Who is responsible for keeping the server secure and up to date?
Your organization, or the managed service provider acting for it, is responsible for host security, OS updates, monitoring, backups, and recovery. The platform provides process isolation, credential controls, and policy gates, but operating system security sits outside its scope.
What is the difference between an on-premise LLM and self-hosted AI agents?
On-premise LLM refers specifically to where model inference runs. Self-hosted AI agents is the broader architecture: the orchestration runtime, task tracking, document storage, memory, channels, and workflows that coordinate agent behavior around that inference layer. This article covers the inference and hardware layer within it, and the architecture page covers the whole.
Where to go next
Decide the inference mode first, because the hardware answer changes by an order of magnitude between the two columns. Set up your node when you are ready to size a deployment against the configuration you actually chose.