Open-source AI agents can still leave the vendor in control

Public code is not enough when a vendor still controls workflow state, permissions, credentials or recovery.

The short version

Open source AI agents require control over models, state, permissions, execution, credentials, traces, and practical exit routes.
Repeated job-level tests expose unstable outcomes: identical TRACE reruns flipped 15–36% compared with the immediately preceding run.
Production buyers should demand checkpointed workflows, code-enforced authority, inspectable traces, independent evaluation, and vendor-free recovery.

The agent opened a customer record, called an internal tool and requested permission to change production. My magical demo became an interrogation about credentials and the 3 a.m. pager.

That mood swing defines my approach to open source AI agents.

I want an actively maintained stack where I control the model, memory, permissions and execution environment. Public code and swappable models help. Bellissimo, until the vendor owns the workflow state or stopping a runaway process requires a support ticket.

The real question is how much of the running system I can inspect, replace and operate without one company’s permission. Source availability may cover orchestration while a hosted control plane owns state. Open weights may carry license restrictions. A self-hosted model may call proprietary tools that make leaving a six-month project.

I deploy in the same order: inspect the stack, run the workflow, restrict its authority, then make the economics survive Tuesday morning.

I draw the exit route first

I draw the system on one page: the model proposes decisions, the harness manages state and tool calls, the runtime enforces permissions, and the workflow stores what it needs later. I circle every irreplaceable box in red and price the exit. This has killed several crushes born from slick demos, probably healthier than my romantic history with SaaS. The best AI app builder for a weekend prototype can become a spectacularly annoying production dependency by Monday. I also test whether workflows export usefully. A folder of JSON spaghetti technically counts, much like airport pesto technically counts as Italian food.

“Open” needs an object. Open weights let me run a model under its license. Open-source orchestration lets me inspect the loop coordinating each step. Operational control means access to the deployment boundary, stored state, tool permissions and audit records. A GitHub repository reassures me about as much as “authentic Italian” printed on airport pesto.

Then I test the hardware bill. In my August 2026 measurements, both gpt-oss models ran fully in memory on an M3 Max with 128GB of unified memory.

The smaller gpt-oss:20b generated about 74 tokens per second, processed prompts at roughly 756 tokens per second and produced its first token in 3.7 seconds. Local inference felt interactive, not like waiting for Trenitalia during a strike.

The gpt-oss:120b generated about 51 tokens per second, processed prompts at around 215 tokens per second and produced its first token in 5.8 seconds. It remained usable for many local workflows, though every call adds latency and another chance to wander into the culinary equivalent of cream in carbonara.

Hardware contention changes everything. My RTX 5060 Ti with 16GB was running ComfyUI, leaving Ollama only 150MB of VRAM and pushing the 20B language model entirely onto the CPU. The card may run my best AI for image generation, but it cannot also power the agent’s language model while ComfyUI occupies its memory. The NVIDIA price-to-earnings ratio tells me little here. The memory monitor does.

I calculate cost per completed job because token prices hide expensive workflow failures. A cheap local model that retries tools, waits on a saturated GPU and needs frequent review can cost more than a pricier model that finishes cleanly, especially after infrastructure and engineer babysitting. Then I repeat the workflow under load. Founder math gets suspiciously creative after one successful demo.

Colourful image tiles crowd a graphics card beside a separate processor labeled language model.

The harness has to survive Tuesday morning

Most agent failures are deeply uncinematic: a worker restarts, a tool returns malformed output, or a laptop sleeps mid-run and wakes ready to repeat an expensive action because nobody recorded it.

State management matters more to me than chat quality. A durable harness creates a run, records completed steps and checkpoints state before continuing. When approval is required, it waits there for a human. If the worker dies, it reloads saved state instead of reconstructing history from vibes. Credentials remain with the harness and appear only when an authorized tool needs them. Every result returns to the workflow for the next model call. This plumbing repels tourists, which is why it matters.

DXC’s OASIS architecture is a useful production example. LangGraph orchestrates agents given company-specific system context plus customer, operational and vendor information. They investigate operational events in parallel and call specialized capabilities. The harness checkpoints each run. Decisions, tool calls and outcomes enter a trace, while credentials stay outside the model. That limits what prompts can expose and leaves operators an inspectable record. DXC says expert-authored evaluations and evidence from successful actions may eventually support wider authority.

LangChain’s Julia Schottenstein described the ambition:

We're proud to bring that model into mission-critical enterprise operations through our partnership with DXC.

“Mission-critical” carries impressive weight. The architecture interests me because the model works inside a process that can pause, resume and show its work. DXC’s material remains first-party evidence, so I would test the same workflow under my own load and failure conditions before handing it anything spicy.

I want traces connecting each outcome to its exact execution. “Agent completed task” is decorative wallpaper. An execution identifier tied to decisions and tool calls lets me reconstruct the route, inspect failures and compare reruns. Without it, debugging feels like interviewing a cat about a broken glass.

Authority belongs in code

I give new agents read-only access. Prompts describe desired behavior; deterministic software decides what may happen.

DXC describes four authority tiers: observation, proposal, human-approved action and action within a defined scope. Middleware evaluates proposed actions by reversibility and impact, then considers whether outcomes are verifiable and how the agent performed previously. Anything above the permitted tier triggers an interrupt before tool execution. The harness checkpoints and waits for someone to approve, edit or reject the request, freezing execution meanwhile. After approval, it can attach the required credential without exposing it to the model. The trace links the request, human decision and outcome to one execution.

This makes policy-defined authority real. Middleware pauses sensitive actions before execution. Because the run is checkpointed, the interrupt creates a clean human decision point. The trace preserves the request and outcome. Evidence from successful actions may eventually justify broader permissions for a narrow task class. Enthusiasm is not evidence, even with tasteful gradients.

Delegated work deserves equal suspicion. A research agent may start with harmless documents, then encounter instructions embedded in a web page or relayed by another agent. Nobody has published enough evidence showing how well these runtime guardrails resist prompt injection or cross-agent compromise in production.

The missing outcome distribution bothers me more than polished averages. I want failure rates, tail resolution times and the frequency with which human intervention prevented harm. I want to know which failures were reversible and which reached a tool unnoticed. DXC has not published that picture.

Benchmarks need reruns and receipts

I test the actual job using normal cases, ambiguous requests, unavailable tools and situations where doing nothing is correct. Whenever possible, I grade the resulting business state: the right record changed, the protected record remained untouched, or the order never went through.

For each run, I freeze the task version and model, then record the framework release, tool interface and permission policy. I repeat identical cases because agents sample actions rather than replaying fixed programs. Then I change one layer while holding the rest steady. Combining a model swap, new tools and a revised grader creates an experiment worthy of a founder explaining churn to investors. I record latency, retries, inference spend and reviewer effort beside the outcome. Failed runs restart from checkpoints so I can locate behavioral changes. Broader authority follows only repeated evidence of bounded failure severity.

TRACE shows why one run proves little. Identical reruns flipped 15–36% of outcomes compared with the prior run. Renaming tools changed scores despite identical underlying operations. Two frontier model judges also disagreed on identical records because they rewarded different parts of a run. Higher scores may reflect presentation, randomness or evaluator preferences rather than improved capability.

Agentic AI coding tools have their own benchmark mess. BenchChecker removed samples it judged contaminated from medium-difficulty bug-fixing tasks. Reported resolution rates then fell by more than 20% for most tested models versus the original results. That is reason enough to inspect a benchmark before using its leaderboard for purchasing, especially when production looks nothing like the test.

DXC reports at least 92% diagnostic accuracy against senior engineer judgment across more than 500 measured OASIS use cases. It also reports a 95% average reduction in incident-resolution time, though it published neither the aggregate earlier baseline nor an independent benchmark.

iTechGuides fairly objects that these are still DXC’s claims without an independent technical assessment. Results across hundreds of use cases deserve attention, especially with enterprise context and human escalation. But attribution remains unanswered: nobody can separate improvements from the model, orchestration harness, tool design or approval policy. Independent workloads and external audits would help.

By 2027, I expect serious procurement reviews to require repeated job-level evaluations and runtime traces before counting GitHub stars. My test remains embarrassingly practical: ask a sleepy engineer to identify the running agent, stop it, inspect its last action and recover the workflow without calling the vendor.

If they need the vendor, the vendor owns the system. I own a repository link.

Frequently asked questions

What makes an AI agent truly open source?

An AI agent is operationally open only when its model, orchestration, workflow state, deployment boundary, permissions, tool access, and audit records can be inspected, replaced, and operated without one vendor’s permission. Public code alone is insufficient when a hosted control plane or proprietary tools make migration impractical.

How should open source AI agents be tested before production?

Open source AI agents should be tested on real jobs using normal cases, ambiguous requests, unavailable tools, and situations where no action is correct. Identical cases require repeated runs because outcomes vary. Evaluations should record business results, latency, retries, inference costs, reviewer effort, model versions, permissions, and tool interfaces.

What safeguards should control an AI agent’s authority?

New AI agents should begin with read-only access, while deterministic middleware controls permitted actions. Sensitive requests should trigger a checkpointed pause for human approval, editing, or rejection before tool execution. Credentials should remain outside the model, and traces should connect every request, decision, tool call, approval, and outcome.