Most teams that want a local AI server discover mid-build that "local" means two different things, and the choice sets the hardware budget before a single line of software is installed. The orchestration layer and your data always live on the machine you control. Model inference is a separate decision, and it is the one that moves the cost by an order of magnitude.
Short answer: A local AI server runs an orchestration platform and your data on hardware you control. Whether AI inference also stays local depends on whether you provision dedicated compute for open-weight models or route reasoning tasks to an external model gateway. The orchestration layer runs on 2 vCPU, 4 GB RAM, and 20 GB storage; fully private inference requires additional CPU or GPU resources sized to the model weights you choose.
What you need before you start
Two configurations are viable, and the one you pick here decides every requirement underneath it.
| Requirement | Hybrid: reasoning through an external gateway | Fully local: open-weight models on the server |
|---|---|---|
| Server hardware | Linux host (Ubuntu or Debian) with 2 vCPU, 4 GB RAM, 20 GB SSD | The same base, plus dedicated CPU or GPU compute sized to the chosen model weights |
| External API credentials | API credentials for the gateway you pick (Anthropic, Google Gemini, OpenAI, or similar) | None for inference |
| Technical administration | Linux server administration, in-house or through an integrator | Linux server administration, in-house or through an integrator |
The hybrid row is a real deployment, not a compromise you settle for later: the runtime, the databases, the documents, and the conversation history sit on your server, and only the prompt context of a given request crosses the egress boundary to a model provider. The fully local row buys a smaller egress boundary and pays for it in hardware. That is the trade, and it is the same trade at every team size.
Steps, in order
- Choose your inference mode. Decide whether agents reason through an external frontier model gateway or through open-weight models running on the server. This fixes hardware spend, credential requirements, and the data boundary for every request that follows. Agents on the same node can be pointed at different models, so a role that handles routine work and a role that drafts contracts do not have to share one setting.
- Provision the server. For hybrid, a virtual private server running Ubuntu or Debian with at least 2 vCPU, 4 GB RAM, and 20 GB SSD. For fully local, add CPU or GPU compute to that base, sized to the model weights you intend to run.
- Install the platform. The self-hosted multi-agent platform runs on Linux, keeps relational state and task records in local PostgreSQL databases, writes system events and agent actions to an append-only event log, and stores documents in a Git-backed repository with version history. The installation runbook covers prerequisites and the install sequence.
- Configure agents and their capability limits. Each agent runs as an isolated process under its own system account, with a declared list of tools it is permitted to call and nothing beyond that list. External messages, record deletions, and privileged system commands stop at a human-in-the-loop gate. For the wider picture of the agent model and what runs on the node, see the self-hosted AI agents overview.
- Load credentials into the secret store. Model gateway API keys, mailbox passwords, and database tokens live in the local secret store. Agents resolve them over a local socket against a strict allowlist, and the values are never placed in a model prompt.
- Connect communication channels. Attach what the business actually uses: the browser office panel, email over SMTP and IMAP, Telegram bots, or a voice interface. Each channel adapter sanitizes input before it reaches an agent, and each conversation maps to a persistent thread.
How to check it worked
- Confirm every agent you configured is running as its own isolated process, visible in the platform panel.
- Run one test task and confirm the task view shows it completed with the full action sequence attached.
- Confirm new entries appeared in the append-only event log after that task ran.
- Trigger an action that needs a human gate, such as an external email or a record deletion, and confirm the system asks for confirmation before it acts.
- Call a tool an agent was not granted and confirm the request is refused rather than executed.
What goes wrong and how to tell
Most post-install trouble traces back to a mismatch between what was configured and what was actually provisioned or scoped. Two of these surface in the first hour, and the third is a decision to make before you buy anything.
Warning: If you chose fully local inference but provisioned only the hybrid minimum (2 vCPU, 4 GB RAM), completions will be slow or will not finish at all. Size compute to the model weights before you deploy agents that depend on local inference, not after.
Warning: If an agent cannot resolve a credential, it returns an error instead of falling back to an insecure path. Check the secret store allowlist first; the agent's tool configuration is usually not the source of that error.
It fits when:
- You want the runtime, task queue, documents, conversation history, and configuration on infrastructure you own or rent.
- You want one node to serve the whole team without per-seat licensing.
- You have Linux administration capability in-house or through an integrator.
- You are prepared to own host security, OS updates, monitoring, backups, and recovery as operational duties.
It does not fit when:
- You expect the software installation itself to satisfy HIPAA, GDPR, or SOC 2. Compliance depends on the full infrastructure configuration, access policies, retention practices, and administrative procedures around it.
- You need a fully air-gapped machine. That requires local model weights and no outbound network adapters, which is a different build from a local install.
- You have no technical staff and no access to an integrator.
Frequently asked questions
Does all the AI inference happen on my server?
Not unless you provision local model weights. The orchestration runtime, the databases, the documents, and the task history stay on your server in every configuration. In a hybrid setup, reasoning tasks are sent to an external frontier model gateway, and the context payload for each request crosses the network to that provider.
What happens to my data when a prompt goes to an external model gateway?
The specific context payload sent in that request travels to the provider and is governed by that provider's commercial API policies. Data held locally in your databases, document repository, and memory store does not leave the server unless a task explicitly includes it in a gateway request. Local storage is never used to train third-party public models.
How much does it cost to run a local AI server?
It depends on the configuration, and there is no zero-cost version of it. A hybrid node runs on modest virtual hardware (2 vCPU, 4 GB RAM, 20 GB SSD) plus whatever the model gateway charges per token. Fully local inference replaces the per-token cost with dedicated CPU or GPU hardware sized to the model. Per-seat licensing is eliminated either way: one node serves the entire team.
Do I need a GPU to run a local AI server?
No, if you route inference to an external frontier model gateway. GPU or dedicated CPU compute is required only when open-weight models run on the machine itself.
Can one server handle my whole team without per-user fees?
Yes. There are no per-seat licensing costs. Capacity is a function of the compute you provision, not the number of people using it.
Next step
Confirm your configuration choice, check the prerequisites, and follow the installation runbook to deploy a node. Set up your node.