Self-hosted AI agents — follow the bytes, not badges

Trace model inference, tool execution, memory and logs to learn what stays local—and when an RTX 5090 or…

The short version

Self-hosted AI agents may keep execution local while sending reasoning, code context and outputs to cloud infrastructure.
An RTX 5090 helps fully local inference only when model weights, runtime overhead and KV cache fit in memory.
Readers should map every processor, storage destination and tool authority before trusting a self-hosted badge or buying hardware.

“Self-hosted” can mean your source code sits downstairs while an AI model in somebody else’s cloud reads it. Lovely linguistic trick.

Give an agent a bug and it looks local: open the repo, edit a file, run tests, return a diff. Underneath, those steps may cross several machines and two companies. Ignore the sticker. Follow the bytes.

Fully local agents keep the model and agent loop inside infrastructure you control. Hybrid systems execute tools near your code while reasoning in the vendor’s cloud. Both are useful. Their security and data boundaries differ sharply.

Karine Brunet said:

Today’s organisations operate in highly interconnected technology ecosystems where complete independence is rarely achievable.

A server rack cannot confer absolution, ragazzi.

Draw the boundary before you trust the badge

Before approving an agent deployment, I draw six boxes: model, harness, worker, memory, logs and enforcement. The model proposes actions. The harness assembles context and runs the loop. The worker edits files, runs commands or drives a browser. Memory preserves state between turns. Logs reveal what happened after the demo-day confidence fades. Enforcement decides whether an action can touch production. If a vendor cannot diagram those boxes, I assume marketing designed the architecture.

Bar chart comparing current figures against their baselines: dual-3090 DFlash2 greedy decode… 474 tok/s versus 432 tok/s, single-stream prose decode on the… 85 tok/s versus 98 tok/s, single-user speculative decode at long… 32 tok/s versus 68 tok/s, steady-state aggregate decode throughput… 1100 tok/s versus 46 tok/s.

Cursor’s Self-Hosted Machines documentation cleanly illustrates the hybrid model. Cursor keeps inference, planning and the agent loop in its cloud. Its CLI worker runs in customer-managed infrastructure, performing file edits, terminal commands, browser actions and local MCP calls. The checkout, build cache and machine-local credentials stay on the worker. Cursor supports up to 200 connected workers per user and 1,000 per team. This is production infrastructure, not a Raspberry Pi behind the router.

One turn works like this. I submit a coding task, and Cursor’s cloud agent requests a file. My worker has opened a long-lived outbound HTTPS connection, so Cursor can send the tool call without initiating an inbound connection. The worker reads the file locally and returns the needed content. Cloud inference consumes it, plans another action and sends the next call through the same connection. The worker may edit the file, run tests and return the diff and terminal output. The loop continues until completion or an external control stops it.

That outbound connection usefully reduces inbound network exposure. It also keeps the session connected to Cursor’s cloud throughout the run.

Team pools add machinery. Cursor’s controller monitors queued requests and calls a customer-supplied spawn mechanism when capacity runs low. An available worker claims a chat and receives its activity. This helps when internal code, strange build images or custom hardware make vendor-managed environments impractical. My team then owns worker images, credentials, patches and scaling. Eventually, somebody gets the 3 a.m. alert when the spawn script develops opinions.

Fully self-hosted AI agents also move inference and orchestration into the customer environment. Coder explicitly distinguishes cloud-hosted reasoning with customer-run execution from systems self-hosting the agent beside its tools. That matters for air-gapped environments and organisations that prohibit cloud inference.

Martin Merz said:

La résilience numérique de l’Europe repose sur une souveraineté à la fois sécurisée et scalable.

“Runs in your VPC” tells me where the shell command happened. I still want to know where the prompt went.

Private execution can still export the session

Hybrid self-hosting has a strong case. Repositories and machine-local credentials can stay on customer hardware, workers can directly reach private services, and outbound connections avoid another exposed inbound port. I would choose it over recreating a delicate internal environment in a generic hosted runner. But I once treated local execution as a proxy for local data handling. Sloppy. Worker location answers only part of the question.

Cursor says runs can send file contents, terminal output, diffs, screenshots, local MCP results and routing metadata to its cloud. Artifacts may also reach Cursor-managed storage. The checkout stays on my machine while selected information derived from it travels elsewhere. A contractual promise never to train on customer data covers one possible use. Inference processing and later storage are separate concerns.

Suppose the worker reads config.production.ts. The file remains on local disk, but some contents enter the context so the model can diagnose a bug. Cloud inference receives that context and proposes a tool call. The worker executes it with my granted permissions, then returns the output so the cloud can plan again. With observability enabled, screenshots or logs may travel through another path. Returned results make the agent more capable because the model sees what happened. They can also contain source code, customer data or a secret printed during debugging.

RTX 5090 under load in an open-sided benchmark chassis, highlighting cooling hardware and isolated power connection in a testing lab.

Security can fail elsewhere. A repository may contain a malicious README, an issue may quote forged instructions, or a browser tool may ingest hostile web text. Once inside the context, that material can influence actions performed with legitimate local credentials. A worker inside my private network gives the agent better access to internal systems. The model must still separate trusted instructions from authoritative-looking text, and models are unreliable bouncers.

Zichuan Li, Jian Cui, Ashley Chen, Xiaojing Liao and Luyi Xing examined a dozen real agent harnesses. Their analysis found context privilege-escalation paths capable of manipulating tool calls, causing denial of service or enabling remote code execution. Jun Wen Leong’s authority-spoofing research found configurations where models recognised forged authority yet still produced the conflicting tool call. Saying “yes, this looks suspicious” before doing the suspicious thing is very on-brand for software.

SkillGuard uses a stronger mechanism. When untrusted external output enters agent state, an inline reference monitor marks the state contaminated and restricts its reachable capabilities. In its AgentDojo evaluation, SkillGuard eliminated attack success on three of four benchmark suites versus an LLM-only baseline. The defence operates outside the model’s judgement, exactly where I want a security boundary. The paper cannot show how well it survives proprietary agents, changing toolsets or adaptive production attackers. Nobody has those independent measurements yet.

For risky tasks, I use disposable workers, narrow credentials, human approval before irreversible actions and egress rules. My convenience instinct remains the attacker’s unpaid intern; “temporary admin access” has outlived several of my startups.

We still lack credible production comparisons between hybrid systems, cloud-managed agents and fully local stacks. Nobody knows how often worker output contains secrets, whether private execution materially reduces prompt-injection success, or how often self-hosted pools fail through neglected patches and credentials. Vendors have architecture diagrams. The incident data has yet to arrive.

An RTX 5090 solves a specific problem

Most personal agents do not need an RTX 5090. A hybrid worker can use an ordinary Linux box or Mac because expensive inference happens elsewhere. Custom hardware may accelerate compilation, rendering or tests, but not a cloud model’s next token.

A fully local agent changes the shopping list. Model weights must fit in fast memory alongside runtime overhead and the KV cache. Longer context uses more cache; concurrent sessions multiply the pressure. Once weights spill into slower memory, interactive coding becomes a meditation retreat. Before buying hardware, I load the exact quantization, replay representative prompts and measure peak residency. Clean benchmark prompts rarely resemble an agent stuffed with six files, terminal output and a regrettable package-lock diff.

The RTX 5090 is extremely fast when the workload fits. In one local-agent benchmark, speculative decoding pushed Qwen’s dense coding model to about 85 tokens per second, versus 74 without speculation. That matters interactively because every tool result starts another generation round.

For unattended loops, the benchmark author chose Gemma’s mixture-of-experts model after measuring 199 tokens per second on that GPU. Interactive coding rewarded Qwen’s behaviour; background automation rewarded Gemma’s speed. Anyone selling one universal “best local LLM” has benchmarked away the workload.

Older cards still work. A controlled test found two RTX 3090 cards delivered 16% to 35% more greedy-decode throughput than one with the model and harness fixed. Tensor parallelism produced the gain, though PCIe communication prevented anything near doubling.

More cards can even reduce single-user speed. In one four-card prose test, generation fell from about 98 tokens per second on an NVLinked pair to 85 because the wider all-reduce crossed PCIe. The larger setup gained enough KV-cache capacity for multiple full-context requests, but each conversation slowed. GPU shopping remains Italian bureaucracy with louder fans.

My M3 Max with 128GB of unified memory ran gpt-oss:20b fully resident at about 74 generation tokens per second. Prompt processing reached roughly 756 tokens per second, with the first token arriving in under four seconds.

On the same machine, gpt-oss:120b generated around 51 tokens per second, versus 74 for the smaller model, and took nearly six seconds to respond. That works for deliberate tasks, but I use the smaller model for chatty coding loops where each tool call creates another turn.

My RTX 5060 Ti contributes nothing to language inference because ComfyUI already owns its memory. Ollama sees roughly 150MB of VRAM, sending a 20B model entirely to the CPU. “Total GPU memory” becomes fan fiction when several home-lab services share a machine.

The best home-lab server is the smallest box that stays available, isolates untrusted jobs and recovers cleanly after failure. Start with the computer on your desk. Add a dedicated GPU when traces prove inference is the bottleneck. A Raspberry Pi works for webhooks or a lightweight worker; I have no defensible Pi inference benchmark and refuse to invent one because the tiny board looks adorable.

The best Ollama model for coding lives inside a loop

My best Ollama coding model is the smallest one that reliably completes tasks from my repositories within the available memory and latency budget. I test failing-test repairs, multi-file refactors and dependency updates requiring real repository search. Model version, quantization, context limit and tools stay fixed. I run each task several times because agents take different paths. Tests judge the patch. A charming explanation beside a broken diff gets the traditional Italian score: no dinner.

The harness may matter more than a leaderboard gap. It selects files and builds the context shaping tool choice. Tool output becomes the next prompt, so one noisy command can derail everything after it. Retry policy turns errors into either recovery or infinite loops. Verification catches elegant patches that break the build. Persistent memory preserves repository conventions—and stale instructions clinging like a nonna’s plastic-container collection. An older model in a clean loop can beat a newer one in chaos.

That speculative-decoding speedup is not a law of physics; it must survive an end-to-end test. It accelerates generation when proposed tokens are accepted often enough, as the RTX 5090 Qwen result showed. The vLLM maintainers have also measured workloads where poor acceptance made multi-token prediction reduce throughput and increase per-token latency. I benchmark the full agent conversation, including long context and tool turns.

By the end of 2027, a “self-hosted” badge without a machine-readable data-flow manifest will look evasive. Every processor, storage destination and tool authority should appear in one inspectable file. Companies still hiding behind the badge will resemble restaurants refusing to show the kitchen.

Buy the GPU later. Demand the data-flow diagram today.

Frequently asked questions

What actually runs locally in self-hosted AI agents?

Fully local agents keep model inference, the agent loop, tool execution, memory, and logs inside infrastructure the operator controls. Hybrid agents can run file edits, commands, tests, browser actions, and local MCP calls on customer-managed workers while cloud systems perform inference, planning, and orchestration.

Do self-hosted AI agents need an RTX 5090?

An RTX 5090 is unnecessary for hybrid agents because expensive model inference happens in the cloud. It becomes useful for fully local agents when the chosen model weights, runtime overhead, and KV cache fit in GPU memory and interactive generation speed materially improves the workload.

What is the best Ollama model for coding?

The best Ollama model for coding is the smallest model that reliably completes representative repository tasks within available memory and latency limits. Evaluation should fix the model version, quantization, context limit, and tools, repeat failing-test repairs and multi-file changes, and let tests judge each patch.