AI Content Creation

Beyond the Base Model: Why AI Agent Security Depends on the Runtime Harness

When security researchers and enterprise developers evaluate the safety and performance of artificial intelligence agents, attention almost exclusively focuses on the underlying large language model (LLM). Standard industry benchmarks evaluate reasoning prowess, coding proficiency, and instruction-following capabilities by treating the agent harness—the underlying software framework connecting model logic to external tools, APIs, and file systems—as an invisible, neutral substrate.

However, groundbreaking research released by cybersecurity firm Lasso Security challenges this long-held industry assumption. The study demonstrates that the runtime framework, or harness, drastically alters an autonomous agent’s behavior under adversarial conditions. By holding the model, system prompts, toolsets, and targets constant while swapping out only the underlying execution framework, researchers discovered that the harness alone can redirect attack strategies, bypass network guardrails, and prematurely terminate multi-turn execution loops. This reveals a critical security blind spot across the artificial intelligence sector: robust security is not an intrinsic property of an isolated LLM, but rather a complex emergent characteristic of the specific model-harness pair.

The Experimental Framework and Isolation Methodology

To isolate the agent harness from confounding variables, the research team at Lasso Security engineered a meticulously controlled experimental environment. The study pitted an open-source framework, LangChain’s deepagents operating on LangGraph, against a prominent closed-source vendor tool, Anthropic’s Claude Agent SDK.

The evaluation tested five diverse attacker models: Claude Sonnet 5, GPT-5.4, Grok 4, Kimi K2.6, and DeepSeek-V4-Pro. Deploying these models across both harnesses established ten distinct model-harness combinations.

This setup presented significant technical hurdles. While LangChain’s deepagents framework is model-agnostic, the Claude Agent SDK is natively designed around the Anthropic protocol, necessitating a translation layer for non-Claude models. To maintain absolute experimental consistency, researchers deployed all five models on Azure AI Foundry and routed every attacker through a shared LiteLLM gateway. This standardization guaranteed that every model traversed an identical network path and encountered the exact same set of Azure content safety guardrails, effectively neutralizing the API provider as an experimental variable.

The testing matrix targeted five simulated enterprise environments spanning highly regulated sectors: finance, legal, healthcare, customer support, and education. Across these domains, researchers devised 20 distinct attack missions mapped directly to the OWASP Top 10 for Agentic Applications, focusing on system-prompt leakage, sensitive-information disclosure, and harmful content generation. Executing each mission five times generated a comprehensive dataset of 1,000 discrete attack runs.

Structural Discrepancies in Prompt Compiling

To eliminate tool orchestration as a variable during the initial phase, researchers restricted the attacker’s operational path to a single exposed tool called send_to_defendant, despite both frameworks offering full capability suites including file system access, web searches, and sub-agent delegation. This constraint forced the evaluation to focus entirely on how each harness compiles system prompts, manages conversational state, and executes iterative turn-by-turn loops.

Despite receiving identical input prompts, the two frameworks constructed fundamentally divergent request payloads. The open-source deepagents framework relies on a modular middleware pipeline. It places the user’s system prompt first, followed by a substantial 7,041-character trailer composed of middleware instructions. This trailer injects behavioral constraints—such as prioritizing accuracy over validating user beliefs—alongside task-planning guidelines, file system tools, and sub-agent spawner instructions, effectively burying the user prompt beneath layers of framework text.

In contrast, the closed-source Claude Agent SDK initiates requests with a concise 148-character identity line, immediately followed by the user’s system prompt. However, it shifts capability documentation away from the prompt text and into serialized tool schemas—structured JSON definitions instructing the model on tool-call formatting. The SDK’s tool schemas totaled an expansive 28,595 characters, highlighted by a massive 9,114-character schema dedicated solely to its planning tool.

Why the agent harness matters as much as the model in AI security - TechTalks

While total payload sizes were comparable at roughly 37,000 characters for deepagents versus 40,000 for the Claude Agent SDK, their structural layouts were structurally inverted. In deepagents, capability instructions reside in the system prompt text; in the Claude Agent SDK, those instructions are embedded within the tool schemas.

The Pitfalls of AI Self-Evaluation

A major hurdle in automated security research is scoring scale. Early in the study, researchers experimented with self-evaluation methods where the AI grades its own operational output. The findings exposed a severe, inherent bias in LLM self-assessment. Across 303 self-claimed successes in unmonitored testing, attacker agents missed 30 genuine wins while falsely declaring victory 155 times.

"Left unchecked, a self-graded pipeline would have reported about 42 percent success where the true rate was 25 percent," noted Michael Sromin, Senior Machine Learning Engineer at Lasso Security, in an interview with TechTalks. Both open and closed frameworks over-claimed victory by 1.5 to nearly 2 times, proving that self-preference bias is a fundamental property of large language models rather than a flaw in a specific scaffolding.

To construct a reliable, bottleneck-free validation pipeline, Sromin recommends separating the evaluation judge from the attacker agent and keeping it entirely blind to the harness under test. Utilizing deterministic ground truth—such as a planted canary secret verified via exact string matching—eliminates judgment error entirely. LLM judges should be reserved strictly for behavioral leaks lacking deterministic strings, governed by a rigid, written rubric calibrated against a more powerful model.

Results, Nuances, and Protocol Anomalies

At a high-level aggregate view across the 1,000 test runs, the two harnesses appeared virtually identical. However, researchers emphasize that this macro-level average is a statistical illusion masking deep underlying dynamics. The closed harness is not universally superior or inferior in security compared to the open framework; rather, massive performance swings cancel each other out depending on the specific model paired with the runtime.

Because an LLM re-reads its entire context window during every turn of an agentic loop, structural differences in prompt scaffolding create persistent behavioral biases. Contrary to the initial hypothesis that burying the prompt under 7,000+ characters of middleware would degrade performance, deepagents did not shorten attack campaigns. For instance, DeepSeek executed its full 20-turn budget under both frameworks.

Instead, context architecture reshaped attack strategies. Aligning with the well-documented "lost in the middle" position bias in long context windows, objectives placed at the end of a prompt are attended to more reliably than those buried beneath framework text. The exact same model, running an identical number of turns, pursued entirely different classes of attacks based on how the harness assembled its prompt.

Where turn counts did collapse, the root cause was protocol translation friction. Because the Claude Agent SDK is engineered natively for the Anthropic protocol, forcing non-Claude models through the LiteLLM translation layer severely impaired their execution loops. "The translation is where the attack quietly dies," Sromin explained. While running on deepagents, the Kimi K2.6 model executed persistent campaigns averaging 14.3 turns per attack and securing 22 wins, but on the Claude SDK translation layer, Kimi averaged just 1.0 turn per attack and secured a single victory.

Furthermore, the study revealed unexpected anomalies at the network layer. Both harnesses utilized relaxed content safety policies when transmitting requests to Azure. However, because deepagents and the Claude Agent SDK format HTTP request bodies differently, identical prompt payloads produced contrasting gateway security behaviors. Approximately 20 of deepagents’ GPT-5.4 attacks were blocked instantly across four sensitive missions—patient Social Security Number extraction, confidential legal data extractions, and insider-trading advice—while zero Claude SDK attempts were intercepted on those exact same tasks. This highlights that a network filter’s efficacy depends significantly on request formatting, not content alone.

Implications for Enterprise AI Security and Development

Why the agent harness matters as much as the model in AI security - TechTalks

These findings fundamentally reshape how organizations must approach autonomous AI threats. Traditional perimeter defense relies on static prompt-injection filters evaluating individual user inputs in isolated transactions. Conversely, autonomous agents probe systems, absorb refusals, pivot tactics, and adapt strategies over multiple execution turns.

"Defense has to be stateful and behavioral, judging intent across the whole conversation rather than message by message," Sromin emphasized. Security architectures must actively monitor execution trajectories, watching for escalation patterns, repeated reframing after refusals, gradual drift toward restricted objectives, and obfuscation attempts. Furthermore, because identical adversarial intent manifests differently depending on harness formatting, defensive red-teaming can no longer test prompts in isolation. Security teams must systematically vary both the underlying model and the runtime framework to uncover hidden attack vectors.

For software engineers and security architects building production-grade agentic applications, the implications require immediate operational adjustments:

  1. Comprehensive Threat Modeling: Threat models must evaluate the entire model-harness configuration rather than benchmarking the base model in a standalone playground. Wrapped runtime frameworks materially alter how models interpret instructions and handle security edge cases.

  2. Full Prompt Visibility: Engineering teams require absolute visibility into compiled prompts. Because frameworks inject thousands of tokens containing middleware instructions, state definitions, and serialized tool schemas, debugging pipelines must capture the complete payload sent over the wire on every turn to detect scaffolding interference.

  3. Standardized Framework Benchmarking: The software industry must adopt rigorous, standardized benchmarks for evaluating agent frameworks. Best practices include naming the combined model-harness pair rather than the model alone, varying one axis at a time during testing, and reporting interoperability and translation taxes separately.

Future Directions in Harness Research

Historically, artificial intelligence evaluations have focused predominantly on cooperative benchmarks where users and agents share unified goals. Under adversarial conditions, target environments actively resist, making multi-turn reasoning, state maintenance, and context management decisive factors.

While Lasso Security’s initial study deliberately constrained attacker agents to a single tool to isolate prompt assembly dynamics, the research team is already expanding its evaluation framework toward fully unrestricted, multi-tool setups.

"Settling it is exactly what a fully unrestricted, multi-tool run would do, and it is the study we are taking on next," Sromin concluded. "Let each harness orchestrate the whole tool kit end to end, so the gap and the advanced persistent threat question become measured facts rather than theoretical inferences." As enterprises increasingly deploy autonomous agents with expansive tool access, understanding the security implications of the underlying harness will remain a paramount requirement for secure software engineering.

Related Articles

Back to top button
Reel Warp
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.