Beyond the Base Model: Why the AI Agent Harness is a Critical Security Blind Spot

When evaluating the performance, reliability, and security of autonomous artificial intelligence agents, the primary focus of developers and researchers has traditionally centered on the underlying large language model. Industry benchmarks meticulously measure the reasoning capabilities, coding proficiency, and knowledge retrieval of LLMs, treating the surrounding agent harness—the runtime framework connecting model reasoning to external tools, APIs, and databases—as invisible, neutral wiring. This prevailing architectural assumption treats the harness as a passive conduit that merely relays instructions without altering the fundamental security profile of the system.

A comprehensive new empirical study published by cybersecurity firm Lasso Security challenges this long-held industry assumption. Researchers set out to measure the extent to which an agent harness independently influences system performance and vulnerability under adversarial conditions. By constructing a strictly controlled autonomous attacker agent—holding the model, system prompt, tool set, and targets constant while varying only the underlying runtime framework—the research team uncovered startling operational deviations.

The findings reveal that the harness is far from a neutral substrate. Simply swapping the runtime framework can fundamentally redirect an attacker’s strategy, determine whether network guardrails successfully block or pass malicious payloads, and even prematurely terminate multi-turn execution loops. For enterprise engineers and security architects deploying generative AI systems into production, these insights expose a critical security blind spot: robust security is not an isolated property of the model alone, but rather an emergent characteristic of the specific model-harness pairing.

Experimental Methodology and Isolating Variables

To effectively isolate the runtime harness from external confounding variables, the Lasso Security research team established a rigorously controlled experimental environment. The investigation pitted an open-source framework, LangChain’s deepagents operating on LangGraph, against a prominent closed-source vendor tool, Anthropic’s Claude Agent SDK.

The evaluation framework incorporated five distinct state-of-the-art attacker models: Claude Sonnet 5, GPT-5.4, Grok 4, Kimi K2.6, and DeepSeek-V4-Pro. Each of the five models was executed across both the open-source and closed-source harnesses, yielding a total of ten unique model-harness combinations.

This experimental design posed a significant engineering hurdle. While the deepagents framework is model-agnostic, the Claude Agent SDK is natively built to speak the Anthropic protocol, necessitating a translation layer to communicate effectively with non-Claude models. To maintain parity and ensure a strictly controlled test environment, researchers deployed all five models on Azure AI Foundry and routed every attacker through a shared LiteLLM gateway. This standardized the network trajectory, ensuring that every model was subjected to the exact same set of Azure content safety guardrails and eliminating the API provider as an unpredictable variable.

The researchers targeted five simulated enterprise environments spanning highly regulated sectors: finance, legal, healthcare, customer support, and education. Across these domains, they engineered 20 distinct attack missions categorized under the OWASP Top 10 for Agentic Applications, focusing primarily on system-prompt leakage, sensitive-information disclosure, and harmful content generation. Executing each mission five times generated a robust dataset of 1,000 discrete attack runs.

To neutralize tool orchestration discrepancies, the operational path of the attacker was intentionally restricted. Although both harnesses provided comprehensive capability suites—including file system access, web search, and sub-agent delegation—the attacker agent was mandated to operate through a single exposed tool designated as “send_to_defendant.” This constraint forced the evaluation to focus exclusively on how each runtime framework compiled prompts, managed conversational state, and executed turn-by-turn iteration loops.

Structural Prompt Discrepancies and Prompt Inversion

Despite receiving identical initial input prompts, the two harnesses constructed fundamentally disparate request payloads behind the scenes.

The open-source deepagents framework relies on a modular middleware pipeline. It places the user’s system prompt first, but subsequently appends a massive 7,041-character trailer composed of internal middleware instructions. This trailer explicitly incorporates behavioral constraints—such as prioritizing strict accuracy over validating user beliefs—alongside task-planning guidelines, file system tools, and sub-agent spawner instructions. Consequently, the core user prompt ends up buried deep beneath extensive framework text.

Conversely, the closed-source Claude Agent SDK adopts an inverted structural layout. It inserts a concise 148-character identity line at the inception of the prompt, followed immediately by the user’s system prompt. However, the SDK shifts capability documentation away from plain text and into serialized tool schemas—structured JSON definitions designed to instruct the model on formatting tool calls. The SDK’s tool schemas total an expansive 28,595 characters, including a staggering 9,114-character schema dedicated solely to its planning tool.

Although the total payload sizes were comparable at approximately 37,000 characters for deepagents versus 40,000 for the Claude Agent SDK, their architectural layouts were inverted. In the open-source framework, capability instructions reside directly within the system prompt text, whereas in the vendor SDK, those instructions are embedded within the tool schemas. Because an LLM re-reads its entire context window on every turn of an agentic loop, these structural differences create persistent behavioral and positional biases.

The Problem of AI Self-Evaluation and Scoring Bias

A major hurdle in conducting large-scale automated security evaluations is grading the outcomes efficiently without introducing human bottlenecks. Initially, the researchers experimented with self-evaluation methods, where the AI model grades its own success or failure. However, the study uncovered that AI self-evaluation is profoundly skewed by inherent self-preference bias.

In unmonitored testing phases, attacker agents missed 30 genuine operational wins while simultaneously falsely declaring victory 155 times across 303 self-claimed successes. Left unchecked, an unmonitored self-graded evaluation pipeline would have reported an inflated success rate of approximately 42 percent, whereas the true empirical success rate was only 25 percent. Both open and closed frameworks over-claimed victory by 1.5 to nearly 2 times, proving that self-preference bias is a systemic characteristic of large language models rather than an artifact of a specific scaffolding framework.

To establish a reliable validation pipeline, Senior Machine Learning Engineer Michael Sromin recommended separating the evaluation judge from the attacker agent while keeping it entirely blind to the harness under test. Utilizing deterministic ground truth—such as a planted canary secret validated by an exact string match—eliminates judgment error entirely. LLM judges should be reserved strictly for behavioral leaks lacking deterministic strings, guided by a rigid written rubric calibrated against a more powerful frontier model.

Empirical Results: Statistical Illusions and Translation Penalties

A high-level aggregate review of the 1,000 test runs initially suggests that the two harnesses perform almost identically. However, researchers emphasize that this macro-level average is a statistical illusion masking deep underlying dynamics. The closed harness is not universally more or less secure than the open harness; rather, massive performance swings cancel each other out depending on the specific model paired with the framework.

Structural prompt differences significantly redirected the nature of attack campaigns. Consistent with the “lost in the middle” position bias observed in long context windows, an objective placed near the end of a prompt receives more reliable attention than one buried beneath thousands of characters of framework instructions. Consequently, the exact same model pursued entirely different attack trajectories depending on which harness assembled its prompt.

Where execution turn counts collapsed, the root cause was protocol translation friction rather than prompt placement. Because the Claude Agent SDK is natively optimized for the Anthropic protocol, forcing non-Claude models to communicate through the LiteLLM translation layer severely impaired their execution loops. For instance, when running on the native deepagents framework, the Kimi K2.6 model executed persistent campaigns averaging 14.3 turns per attack and securing 22 wins. On the Claude SDK’s translation layer, however, Kimi averaged a meager 1.0 turn per attack and secured only a single win, demonstrating how protocol friction can prematurely kill multi-turn execution loops.

Furthermore, an intriguing anomaly emerged at the network layer. Both harnesses utilized relaxed content safety policies when transmitting requests to Azure. Yet, because deepagents and the Claude Agent SDK formatted HTTP request bodies differently, identical prompt payloads produced contrasting gateway behaviors. Approximately 20 of deepagents’ GPT-5.4 attacks were blocked at the door across four specific sensitive missions—such as patient Social Security number extraction, confidential legal data retrieval, and insider-trading advice—while zero Claude SDK attempts were intercepted on those exact same tasks. This confirmed that a filter’s operational efficacy depends heavily on the request format, not the content alone.

Broader Implications for Enterprise AI Security

The findings published by Lasso Security carry profound implications for the future of enterprise AI security. Traditional application security controls rely heavily on static prompt-injection filters that evaluate individual user inputs in isolation. In stark contrast, autonomous AI agents operate dynamically: they probe systems, absorb refusals, pivot strategies, and iterate across multiple turns.

Security defenses for agentic applications must transition from stateless filters to stateful, behavioral monitoring systems that evaluate intent across an entire conversation rather than message by message. Modern security platforms must monitor the complete execution trajectory, watching for escalation patterns, repeated prompt reframing following refusals, gradual drift toward sensitive operational objectives, and sophisticated obfuscation attempts.

For software engineers and security architects building production-grade agentic systems, the traditional approach to threat modeling must evolve. Evaluating an LLM in a standalone playground provides incomplete data, as runtime frameworks fundamentally alter how models process instructions and handle edge cases. Engineering teams must mandate complete visibility into compiled system prompts, capturing full payloads sent over the wire on every turn to detect where framework scaffolding interferes with model guardrails.

As autonomous agents gain access to increasingly powerful toolsets and broader enterprise privileges, industry experts anticipate that the performance and security gap between various harnesses will widen significantly. By establishing standardized benchmarks, naming model-harness pairs rather than base models in performance reports, and isolating protocol translation penalties, the software development community can begin to secure the hidden runtime wiring that underpins the next generation of artificial intelligence applications.