Running your own AI assistant sounds straightforward until you try to answer two questions: what actually stays on your server, and what does it take to get something working.
Short answer: A self-hosted AI assistant runs its orchestration runtime, task queue, conversation history, and configuration database on a Linux server you provision and control. When a task calls a frontier model such as Anthropic Claude or OpenAI GPT, the context payload for that request travels to the provider's API; storage and workflow state remain on your server. A node that routes inference to external gateways needs at least 2 vCPU, 4 GB RAM, and 20 GB SSD on Ubuntu or Debian. For the wider category, see self-hosted AI agents.
What you need before you start
Three decisions come before installation: how much server you provision, whether inference runs on external model gateways or on local model weights, and who administers the Linux host. The third is the one teams skip.
| Decision | What it means in practice |
|---|---|
| Server hardware | 2 vCPU, 4 GB RAM, and 20 GB SSD is the floor when inference runs on remote model gateways. |
| Operating system | Ubuntu or Debian. No other distribution is supported. |
| Inference model choice | External frontier gateways, local open-weight models, or a mix across agents on the same node. |
| Local inference compute | If any agent runs local weights, the host must be sized to those weights. No single figure covers every model. |
| Credentials in hand | API keys for the gateways you use, plus mailbox and service tokens, before configuration begins. |
| Technical administration | Someone who can patch, monitor, back up, and recover a Linux host, or an integrator who does. |
The inference decision is the one to make first, because it sets both the hardware bill and the scope of data that leaves the server. A node that only calls external gateways runs on modest virtual hardware. A node that runs local weights needs dedicated CPU or GPU capacity sized to the model. Neither path charges per seat: one node serves an entire team, and what remains is infrastructure cost, maintenance time, and per-token gateway usage.
Steps
- Provision the server before anything else. If any agent will run local model weights, size the host to those weights now. Adding GPU capacity after installation is the most common rework. Otherwise the 2 vCPU, 4 GB RAM, 20 GB SSD floor applies.
- Install the Smart Node runtime. The installation runbook at docs/install lists the prerequisites and the commands for Ubuntu and Debian.
- Configure inference routing. Declare which agents call external model gateways and which run local weights. Egress to external gateways is explicit, metered, and controlled in system configuration, so this step is where the hybrid inference boundary is drawn and where you decide what may leave the server.
- Load credentials into the local secret store. API keys, mailbox passwords, and service tokens go in as references. Agents resolve them at execution time over local authenticated channels under a strict allowlist, and the values are never injected into model prompt context.
- Define agent roles with declared scopes. Give each role the least-privilege set of tools it needs. Each agent runs as an isolated process under an explicit capability allowlist, which is where bounded autonomy is configured rather than assumed.
- Connect the channels the team already uses. Browser-based web chat, internal agent mail, SMTP/IMAP email, Telegram bots, or voice. Channel adapters sanitize inbound input and map each conversation to a persistent thread.
- Set the human-in-the-loop gates. Sending external email, publishing content, deleting records, running privileged shell commands, and spending money each require explicit human confirmation before execution. Treat these as settings you choose, not defaults you inherit.
- Run one task end to end and read the event log. Every system event, state change, and agent action is written to an append-only event log. If a step is missing from the log, the configuration is not finished.
How to check it worked
Run these four checks after the first installation:
- Give one agent a small task and confirm it completes and reports back.
- Trigger a test external action, an email to your own address, and confirm the human-in-the-loop gate stops it and asks for confirmation before anything is sent.
- Open the event log and confirm the task, its state changes, and each tool call appear in order.
- Inspect a model request and confirm no API key, password, or token value appears in the prompt context.
What goes wrong and how to tell
Warning: A hybrid inference setup is not air-gapped. When a task routes to an external frontier model, the context payload for that specific request leaves your server and reaches the provider. That transit is explicit and metered, and it is governed by the provider's commercial zero-retention API policy, but it is not the same as a deployment where nothing leaves the server. That level of isolation requires exclusively local model weights and no outbound network adapters.
Underestimated compute. If a local model runner is configured but responses slow sharply under load, or jobs queue and never finish, the host is not sized to the model weights. The 2 vCPU, 4 GB RAM figure applies only to nodes that call remote gateways.
Neglected host maintenance. Process isolation and bounded agent permissions keep agents out of each other's work. They do not patch the operating system, watch disk usage, or restore a backup. Host security, OS updates, monitoring, backups, and recovery are the operator's responsibility, and an unpatched server undermines every layer above it.
Compliance assumed from installation. Installing the software does not produce HIPAA, GDPR, or SOC2 compliance. Compliance depends on how the infrastructure is configured, who can reach it, how long data is retained, and what administrative procedures surround it.
Frequently asked questions
Does self-hosting mean my data never leaves my server?
No, not by default. Storage and workflow state stay on your server: the orchestration runtime, task queue, documents, conversation history, and configuration database run on infrastructure you own or rent. But when a task routes to an external frontier model, the context payload for that request travels to the provider's API. Only a deployment configured with exclusively local model weights and no outbound network adapters keeps everything inside.
What server hardware do I need for a self-hosted AI assistant?
At minimum, 2 vCPU, 4 GB RAM, and 20 GB SSD on Ubuntu or Debian when inference runs on remote model gateways. If any agent runs local open-weight models on the host, that minimum does not apply: the CPU or GPU has to be sized to the specific model weights you choose.
Can my whole team use one node without paying per user?
Yes. There is no per-seat licensing, so a single node serves an entire team. What remains is the infrastructure cost of the server, the maintenance time that server needs, and per-token charges from whichever external model gateways you route tasks to.
Do I need a developer to install and maintain a self-hosted AI assistant?
Yes, at least for the host itself. Deploying and administering a Linux server takes technical proficiency, or support from an integrator. The platform handles agent isolation, credential handling, and workflow state, but host security, OS updates, monitoring, and backups are yours.
What happens to my API keys and external service credentials in this setup?
They stay in the local secret store on your server. An agent resolves a credential reference at the moment a tool runs, over local authenticated channels and under a strict allowlist, and the value is never injected into model prompt context. A model sees the result of an action, not the key that authorized it.
Does self-hosting automatically make my AI setup GDPR or HIPAA compliant?
No. Installing the software produces no compliance on its own. Compliance depends on the whole configuration: infrastructure setup, access policies, data retention practices, and administrative procedures. Those are operator decisions, and a self-hosted deployment gives you the control to make them.
Deploy a private node
If the sequence above matches what your team needs, the next step is provisioning the server and running the installation. Get started covers prerequisites, the deployment path, and what to configure first.