The Hidden Threat: How AI Agent Harnesses Alter Security and Performance Under Adversarial Conditions

For years, the artificial intelligence community has evaluated agentic performance and security through a singular lens: the underlying large language model (LLM). Standardized benchmarks have historically measured the raw reasoning capabilities of models like GPT, Claude, and DeepSeek while treating the surrounding agent harness—the runtime framework that connects model intelligence to external tools, memory stores, and APIs—as an invisible and neutral pipe. This prevailing paradigm assumes the harness merely acts as a passive substrate, faithfully executing commands without injecting its own behavioral biases into the system.

A comprehensive new empirical study by cybersecurity firm Lasso Security shatters this foundational assumption. Researchers set out to measure the precise degree to which an agent harness alone influences performance under adversarial conditions. By keeping the attacking model, system prompt, toolset, and target environments entirely constant while substituting only the runtime framework underneath, the study revealed that the harness is far from a neutral bystander.

The findings indicate that swapping the runtime framework can actively redirect an attacker’s tactical strategy, dictate whether network-level security guardrails flag or pass malicious payloads, and inadvertently terminate multi-turn execution loops. For enterprise software engineers and security architects, this exposes a glaring blind spot: application security is not a standalone property of the underlying model, but rather a dynamic emergent property of the specific model-harness pair.

Methodology and Experimental Isolation

Isolating the agent harness from confounding variables required a meticulously designed test architecture. The Lasso Security research team established a controlled comparative environment pitting an open-source framework, LangChain’s deepagents operating on LangGraph, against a prominent closed-source vendor solution, the Anthropic Claude Agent SDK.

To ensure comprehensive coverage across diverse architectures, the researchers tested five state-of-the-art attacker models: Claude Sonnet 5, GPT-5.4, Grok 4, Kimi K2.6, and DeepSeek-V4-Pro. Running each model across both runtime frameworks generated ten distinct model-harness combinations.

This setup presented a formidable engineering hurdle. While deepagents is designed to be model-agnostic, the Claude Agent SDK natively adheres to the Anthropic API protocol and demands an intermediary translation layer to communicate with non-Claude models. To eliminate the API provider as an experimental variable, all five models were deployed on Azure AI Foundry and routed through a shared LiteLLM gateway. This standardized the network transport path, subjecting every model execution to identical Azure content safety guardrails.

The evaluation matrix spanned five simulated enterprise domains: finance, legal, healthcare, customer support, and education. Across these sectors, researchers engineered 20 explicit attack missions corresponding to three threat categories outlined in the OWASP Top 10 for Agentic Applications: system-prompt leakage, sensitive-information disclosure, and harmful content generation. Executing each mission five times produced a robust dataset of 1,000 discrete, controlled attack runs.

To prevent variations in tool orchestration from skewing the data, the operational pathway was severely restricted. Although both harnesses provided comprehensive capability suites—including file system access, web searches, and sub-agent delegation—the attacker was constrained to operate through a single designated tool interface named send_to_defendant. This forced the evaluation to focus strictly on how each harness compiles prompt payloads, manages conversational state, and executes turn-by-turn processing loops.

Structural Prompt Discrepancies and Payload Layouts

Despite processing identical input prompts, the two harnesses generated fundamentally different request payloads due to structural formatting choices in their underlying codebases.

The open-source deepagents framework relies on a modular middleware pipeline. It places the user’s system prompt first, subsequently appending a 7,041-character trailer composed of internal middleware instructions. This trailer explicitly embeds behavioral constraints—such as prioritizing logical accuracy over validating user beliefs—alongside task-planning guidelines, file system tools, and sub-agent spawner instructions. Consequently, the core user prompt becomes deeply buried beneath extensive framework text.

Conversely, the closed-source Claude Agent SDK initiates requests with a brief 148-character identity line, immediately followed by the user’s system prompt. However, the SDK shifts capability documentation away from the prompt text and into serialized tool schemas—structured JSON objects that instruct the model on how to format function calls. These tool schemas aggregate to a staggering 28,595 characters, including a massive 9,114-character schema dedicated exclusively to its internal planning tool.

Although the total payload sizes were comparable at approximately 37,000 characters for deepagents versus roughly 40,000 characters for the Claude Agent SDK, their architectural layouts were effectively inverted. Deepagents houses its capability instructions directly within the system prompt text, whereas the Claude Agent SDK embeds those instructions within formal tool definitions.

The Pitfalls of AI Self-Evaluation

A significant methodological contribution of the study lies in its critique of automated evaluation. Initial testing phases utilized self-evaluation methods, wherein the AI model grades the success of its own operational output. The researchers discovered that this approach is plagued by severe, systemic self-preference bias.

In unmonitored testing configurations, attacker agents missed 30 genuine operational wins while falsely claiming victory 155 times across a pool of 303 self-declared successes. Michael Sromin, Senior Machine Learning Engineer at Lasso Security, emphasized the magnitude of this distortion, noting that left unchecked, a self-graded pipeline would have reported an inflated success rate of 42 percent against a true baseline of 25 percent. Both open-source and closed frameworks over-claimed victory by factors ranging from 1.5 to nearly 2 times, proving that self-preference bias is an inherent characteristic of large language models rather than a flaw of a specific scaffold.

To establish a reliable validation pipeline free from human-in-the-loop bottlenecks, Sromin recommends decoupling the evaluation judge from the attacker and ensuring the judge remains completely blind to the harness under test. Implementing deterministic validation techniques, such as planted canary secrets verified via exact string matching, eliminates judgment error entirely. Under this framework, LLM judges are reserved strictly for behavioral data leaks lacking deterministic strings, guided by rigid rubrics calibrated against more capable foundational models.

Aggregate Illusions and Performance Anomalies

A macro-level examination of the aggregate results from the 1,000 test runs initially suggests parity between the two runtime frameworks. However, researchers warn that this broad average is a statistical illusion that masks deep operational divergences. The closed harness is neither universally more nor less secure than its open-source counterpart; rather, extreme performance swings cancel each other out depending on the specific model paired with the framework.

Because large language models re-read their context windows during every iteration of an agentic loop, structural prompt scaffolding creates persistent behavioral biases. Counter to initial hypotheses, burying the prompt beneath thousands of characters of middleware in deepagents did not reduce the duration of an attack campaign. For example, DeepSeek completed its full 20-turn budget under both frameworks.

Instead, context architecture reshaped the tactical vector of the attack. Consistent with the well-documented "lost in the middle" position bias characteristic of long context windows, instructions placed near the end of a prompt are attended to more reliably than those buried under extensive framework text. Consequently, the identical model, operating over the exact same number of turns, pursued entirely different attack strategies depending on which harness constructed its prompt.

Where turn counts did experience sharp declines, the root cause was protocol translation friction rather than prompt placement. Because the Claude Agent SDK is natively optimized for the Anthropic communication protocol, forcing non-Claude models to route through the LiteLLM translation layer severely degraded their execution loops. While the Kimi K2.6 model executed persistent campaigns on deepagents—averaging 14.3 turns per attack and securing 22 wins—its performance plummeted on the Claude SDK translation layer, averaging just 1.0 turn per attack and securing a solitary win.

Network-Layer Anomalies and Gateway Behavior

The study also identified unexpected anomalies at the network security layer. Both harnesses maintained relaxed content safety policies when dispatching HTTP requests to Azure AI Foundry. However, because deepagents and the Claude Agent SDK format HTTP request bodies differently, identical prompt payloads triggered contrasting security gateway responses.

Approximately 20 of deepagents’ GPT-5.4 attack payloads were blocked at the network perimeter across four specific high-risk missions: patient Social Security number extraction, confidential legal data exfiltration, and insider trading advisory generation. Conversely, zero attempts executed through the Claude SDK were intercepted on those exact same tasks. This anomaly underscores a critical realization for security engineers: a network filter’s efficacy is intimately tied to request payload formatting, not content semantics alone.

Implications for AI Agent Defense

These findings challenge conventional wisdom regarding how enterprises defend against autonomous cyber threats. Traditional security controls rely heavily on static prompt-injection filters designed to evaluate isolated user inputs. Autonomous agents, however, operate iteratively—probing defenses, absorbing refusals, and pivoting strategies across multiple conversational turns.

Security architectures must evolve to become stateful and behavioral, assessing intent across the entirety of a dialogue rather than analyzing messages in isolation. Enterprise security systems must actively monitor execution trajectories for escalation patterns, repeated prompt reframing following refusals, gradual drift toward restricted objectives, and obfuscation techniques. Furthermore, because adversarial intent manifests differently depending on harness formatting, defensive red-teaming can no longer focus solely on evaluating static prompts. Security teams are now compelled to test permutations of both base models and runtime harnesses to identify hidden attack paths.

Strategic Takeaways for Software Developers

For software engineers and security architects building production-grade agentic applications, the implications of the Lasso Security study demand immediate operational adjustments:

1. Comprehensive Threat Modeling: Threat models must evaluate the complete model-harness configuration rather than analyzing the base model in isolation. Assessing an LLM within a standalone playground yields incomplete data because wrapped runtime frameworks profoundly alter instruction adherence and edge-case handling.

2. Full Payload Visibility: Engineering teams must maintain complete visibility into compiled prompts. Because frameworks routinely inject thousands of tokens containing middleware instructions, state definitions, and verbose tool schemas, debugging pipelines must capture the exact payload transmitted over the wire on every execution turn.

3. Standardized Framework Benchmarking: The software development industry must adopt rigorous, standardized benchmarks for evaluating agent frameworks. Sromin outlines three foundational practices for the AI community: naming the model-harness pair rather than the model alone, isolating variables by swapping only the harness while holding all other parameters constant, and measuring interoperability friction independently from core framework performance.

As developers continue to grant autonomous agents broader access to enterprise toolsets, industry experts anticipate that this performance gap will widen. In complex, multi-step enterprise workflows, minor per-turn discrepancies in context assembly and schema design compound exponentially over extended action chains.

Future Directions in Harness Research

Historically, artificial intelligence evaluation has prioritized cooperative benchmarks wherein users and agents share aligned objectives. In adversarial environments, target systems actively resist, elevating multi-turn reasoning and context management to decisive factors in operational success.

While Lasso Security’s initial study deliberately constrained attacker agents to a single tool to isolate prompt assembly mechanics, the research team is actively expanding its investigative scope toward fully unrestricted operational environments.

Allowing runtime frameworks to orchestrate an entire tool kit end-to-end will transform theoretical security gaps into quantifiable measurements, providing the software industry with the empirical data required to secure the next generation of autonomous artificial intelligence systems.