Running self-hosted AI models does not mean every inference call stays on your server. It means you decide where each call goes, and why.
Short answer: A self-hosted multi-agent node has two inference paths. Open-weight models run entirely on your own CPU or GPU, so the inference data stays on the server, but they require compute sized to the model weights you choose. Frontier model gateways (Anthropic Claude, Google Gemini, OpenAI) handle complex reasoning, and the context payload for each request transits the network to that provider under its commercial zero-retention API policy.
How self-hosted inference actually works
| Aspect | Local open-weight model | Frontier gateway model |
|---|---|---|
| Data leaves the server | No | Yes, per the provider's API terms |
| Hardware floor | CPU or GPU sized to the model weights | 2 vCPU, 4 GB RAM, 20 GB SSD |
| Best-fit tasks | Routine classification, extraction, structured output | Complex reasoning, multi-step planning, open-ended generation |
| Cost structure | Infrastructure only | Infrastructure plus per-token gateway fees |
| Data policy | Entirely inside your infrastructure | The provider's commercial zero-retention API policy |
Self-hosting describes where the platform runs, not where every inference call goes. The orchestration runtime, task queue, documents, conversation history and configuration database sit on infrastructure you own or rent. The model that answers a given request may sit there too, or may sit behind a commercial API. Which one applies is a configuration decision you make per agent role, not a property of the installation.
A single node holds both paths at the same time. One agent can run an open-weight model on the host while another agent in the same workflow sends its reasoning step to a frontier gateway. Egress to external gateways is explicit, metered and controlled by system configuration: the node does not silently route a task outbound. That gives you a written answer to the question of which roles touch an external provider. For the wider architecture those roles sit in, see self-hosted AI agents.
The trade-off is worth stating before the steps. Unless you provision dedicated GPU hardware and run only local weights, the complex reasoning your agents do will cross the egress boundary on its way to a provider.
A node running both paths at once
A twelve-person operations team runs one node with three agent roles. The intake agent reads inbound email, classifies each message and extracts order references. That is structured output against a fixed category set, so it runs an open-weight model on the host and its input stays on the server. The project agent decomposes multi-week work into tasks and drafts long proposals. That is open-ended generation, so it routes to a frontier gateway, and the context payload for each of those calls crosses the egress boundary. The reporting agent reads the append-only event log and writes a weekly summary of what the other two did; it runs locally, because its input is internal operational history.
The split is visible in configuration, not implied. Because the egress boundary is set at the system level, the organization can name which agent roles touch an external provider and which do not. Adding a fourth role later means making the same decision again for that role, not re-architecting the node.
Human-in-the-loop gates sit on a separate layer from model routing. Sending a drafted proposal to a customer, deleting a record, running a privileged command or committing monetary spend all require explicit human confirmation, whichever inference path produced the text. Model choice changes where computation happens. It does not change what the node is allowed to do without approval.
Steps to decide which path each task needs
- List every agent role you plan to run and describe the task type each one handles: classification, extraction, generation, planning, customer interaction, and so on. You finish with an inventory rather than a decision.
- Rate each task type on reasoning complexity. Routine extraction, classification and structured lookups are low complexity. Multi-step planning, long-form generation and open-ended question answering are high complexity. You finish with a complexity label for every task type.
- Rate each task type on data sensitivity. Ask whether the input contains anything that must not leave the server under any condition: personally identifiable data, regulated records, trade secrets. A yes forces the local path at any complexity level.
- Map each task to a path using the two ratings. High sensitivity requires a local model with adequate compute provisioned. Low sensitivity with high complexity is the primary use case for a frontier gateway. Low sensitivity with low complexity fits either path, so decide that one on cost.
- Audit the hardware against both paths. Gateway-routed agents operate at the 2 vCPU, 4 GB RAM, 20 GB SSD floor. Locally-inferred agents need CPU or GPU compute sized to the weights of the model chosen for that role, which is a much larger commitment than the floor.
- Configure each agent with its designated path. Gateway egress is explicit and metered, and the system does not silently route a local task outbound. You finish with a node where every role has a written answer to the question of whether its data leaves the server.
Common mistakes when choosing a self-hosted model path
Three errors account for most of the bad outcomes here. Each comes from an assumption that sounds right until you check it against the architecture.
Self-hosting is often read as a guarantee that inference happens locally. It is not.
Warning: Unless every agent is configured with a local open-weight model and no external gateway is connected, context payloads for frontier requests leave the server. Self-hosting on its own does not mean local inference.
The second error is hardware sizing, and it usually shows up late, after the roles are configured.
Warning: A 2 vCPU, 4 GB VPS handles gateway-routed operation, not meaningful local inference. Provisioning a local path on the minimum instance and expecting frontier-level output is the most common hardware planning error. Fully private, high-accuracy local models require substantial server hardware or dedicated GPUs.
The third is treating installation as a compliance event.
Warning: Installing the software does not make an organization HIPAA, GDPR or SOC 2 compliant. Compliance depends on the infrastructure configuration, the access policies, the data retention practices and the administrative procedures around it.
Frequently asked questions
Can I run Claude or GPT locally on my own server?
No. Frontier models from Anthropic, Google and OpenAI are available only through their commercial API gateways and are not distributed as local model weights. Local inference means open-weight models running on your own CPU or GPU.
What hardware do I need to run models locally with no external API calls?
The compute requirement scales with the model weights you choose, and the standard VPS floor of 2 vCPU and 4 GB RAM is not enough for meaningful local inference. That floor covers gateway-routed operation. A local path needs dedicated CPU or GPU hardware sized to the chosen model. Because the requirement is model-specific, this page gives no fixed GPU or VRAM figure.
If I use a frontier gateway, who can see my data?
The specific context payload for that request transits to the provider and is governed by that provider's commercial zero-retention API policy. Everything else stays: the orchestration runtime, task queue, documents, conversation history and configuration database remain on your infrastructure. The security page covers where those boundaries sit.
Can different agents on the same node use different models?
Yes. Each agent is configured independently. One agent can run a local open-weight model while another routes the planning step of the same session to a frontier gateway, and the node tracks which is which.
Does using an external model gateway mean I am not self-hosting?
No. The orchestration runtime, task state, documents and all stored data remain exclusively on your infrastructure. The inference call for a complex task is the only boundary that crosses to an external provider, and it is an explicit, metered egress rather than a delegation of the platform itself.
Next step
Review the full architecture against your own agent inventory, then deploy a node: Set up your node.