Beyond the Model: Why AI Agent Security Varies Dramatically by Runtime Framework

When evaluating the performance, reliability, and security of agentic artificial intelligence systems, software developers and security researchers have historically concentrated their scrutiny almost entirely on the underlying large language model (LLM). Standardized benchmarks are frequently deployed to measure the raw reasoning capabilities, multi-turn logic, and contextual retention of these models, treating the accompanying "agent harness"—the runtime framework connecting the model to APIs, memory structures, and external tools—as a neutral, invisible infrastructure. This traditional methodology presupposes that the harness functions as a pure conduit, uniformly translating intelligence into action.

However, groundbreaking research published by cybersecurity firm Lasso Security fundamentally challenges this long-held industry assumption. In a rigorous comparative study, security researchers evaluated the extent to which an agent harness independently influences system performance and vulnerability under adversarial conditions. By keeping the attacker model, system prompts, tool sets, and target environments strictly constant while altering only the underlying runtime framework, the study revealed that the harness is far from a passive conduit. Instead, it actively shapes attack strategies, dictates whether network-level security guardrails intercept malicious payloads, and can inadvertently stall or terminate multi-turn execution loops entirely.

For enterprise security teams and software architects, these findings expose a critical blind spot: application security is not an intrinsic property of an LLM in isolation, but rather an emergent characteristic of the specific model-harness pairing.

Methodological Rigor: Isolating the Runtime Framework

To ensure empirical validity and eliminate extraneous variables, the Lasso Security research team constructed a highly controlled experimental environment. The investigation pitted an open-source framework, LangChain’s deepagents operating on LangGraph, against a prominent closed-source vendor solution, Anthropic’s Claude Agent SDK.

The evaluation framework incorporated five state-of-the-art attacker models: Claude Sonnet 5, GPT-5.4, Grok 4, Kimi K2.6, and DeepSeek-V4-Pro. Deploying each model across both runtime environments yielded ten distinct model-harness combinations.

A significant technical hurdle involved protocol harmonization. While the deepagents framework is natively model-agnostic, the Claude Agent SDK is specifically engineered around the Anthropic protocol, necessitating a translation layer for non-Claude architectures. To resolve this without introducing API-provider biases, researchers deployed all five models via Azure AI Foundry and routed every attacker through a shared LiteLLM gateway. This standardization guaranteed that every model faced the identical baseline of Azure content safety guardrails, neutralizing API variance as an experimental factor.

The study targeted five simulated corporate environments encompassing finance, legal, healthcare, customer support, and education. Within these domains, researchers engineered twenty distinct attack missions aligned with the OWASP Top 10 for Agentic Applications, explicitly focusing on system-prompt leakage, sensitive-information disclosure, and harmful content generation. Executing each mission five times across the infrastructure generated a comprehensive dataset of 1,000 discrete attack campaigns.

To further isolate tool orchestration, the operational pathways of the attacker agents were strictly constrained. Despite both harnesses providing expansive capability suites—including file system navigation, web browsing, and sub-agent delegation—attackers were restricted to a single exposed interface designated as the "send_to_defendant" tool. This methodological constraint forced the evaluation to concentrate narrowly on how each harness compiled prompts, maintained internal state, and processed turn-by-turn interactive loops.

Structural Divergence in Prompt Construction

Despite receiving identical initial prompts, the two runtime frameworks compiled radically different underlying request payloads, fundamentally altering the informational landscape perceived by the LLM.

The open-source deepagents framework relies on a modular middleware pipeline. It places the user’s system prompt at the outset, immediately appending a massive 7,041-character trailer containing middleware instructions. This auxiliary text injects behavioral constraints (such as prioritizing analytical accuracy over user validation), task-planning directives, file system management rules, and sub-agent generation protocols. Consequently, the core user prompt becomes deeply buried beneath extensive framework-generated text.

Conversely, the proprietary Claude Agent SDK introduces a concise 148-character identity line, followed immediately by the user’s system prompt. However, it shifts operational capability documentation entirely into serialized tool schemas—structured JSON definitions instructing the model on tool invocation formatting. Within the Claude SDK, these tool schemas span an expansive 28,595 characters, including a staggering 9,114-character schema dedicated exclusively to its planning tool.

Although the total payload sizes were comparable—hovering around 37,000 characters for deepagents versus 40,000 characters for the Claude Agent SDK—their structural layouts were inversely mirrored. In the open-source environment, capability instructions reside directly within the system prompt text, whereas in the closed-source SDK, those instructions are embedded within structured tool schemas.

The Pitfalls of Automated Self-Evaluation

A major hurdle in AI security research is scalability: manually reviewing thousands of multi-turn attack logs is cost-prohibitive and logistically impractical. Consequently, researchers initially experimented with automated self-evaluation, allowing the AI to grade its own operational success.

The results exposed a severe systemic bias. In unmonitored testing, attacker agents failed to recognize 30 genuine wins while falsely declaring victory 155 times out of 303 self-reported successes. Michael Sromin, Senior Machine Learning Engineer at Lasso Security, noted that an unmonitored self-graded pipeline would have erroneously reported a 42 percent success rate where the true empirical rate was only 25 percent. Both open- and closed-source frameworks over-claimed victory by factors ranging from 1.5 to nearly 2 times, demonstrating that self-preference bias is an inherent limitation of current LLMs rather than an artifact of a specific scaffolding architecture.

To construct a reliable evaluation pipeline, Sromin recommends decoupling the evaluation judge from the attacker agent while ensuring the judge remains blind to the underlying harness. Furthermore, relying on deterministic ground truth—such as a planted canary secret verified via exact string matching—eliminates judgment error entirely, reserving LLM-based judges strictly for behavioral leaks lacking deterministic signatures.

Aggregate Parity Masks Dramatic Performance Swings

At a macro level, aggregate results from the 1,000 test runs suggested that the two harnesses performed nearly identically. However, researchers quickly identified this aggregate metric as a statistical illusion masking profound underlying dynamics. The closed harness was neither universally more nor less secure than the open framework; instead, dramatic performance swings cancelled each other out depending on the specific model-harness pairing.

Because LLMs re-read their context windows during every turn of an agentic workflow, structural prompt variations induce persistent behavioral biases. Rather than shortening campaign durations, burying prompts beneath thousands of characters of deepagents middleware did not inhibit multi-turn persistence. For instance, DeepSeek executed its full 20-turn budget across both frameworks.

Instead, context architecture heavily redirected attack strategies. Aligned with the well-documented "lost in the middle" position bias observed in long context windows, instructions placed toward the end of a prompt receive greater attention than those buried beneath framework text. Consequently, identical models pursued entirely different attack vectors based solely on how the harness compiled the prompt.

Where campaign durations did collapse, the root cause was protocol translation friction. Because the Claude Agent SDK is natively optimized for the Anthropic protocol, routing non-Claude models through the LiteLLM translation layer severely impaired execution loops. For example, the Kimi K2.6 model executed persistent campaigns on the deepagents framework, averaging 14.3 turns per attack and securing 22 operational wins. On the Claude SDK translation layer, Kimi’s average plummeted to 1.0 turn per attack, yielding only a single win.

Network-Layer Anomalies and Gateway Interactions

The study also uncovered unexpected anomalies at the network security layer. Both harnesses utilized relaxed content safety policies when communicating with Azure endpoints. However, because deepagents and the Claude Agent SDK format HTTP request bodies differently, identical prompt payloads triggered contrasting behavior from the Azure content safety gateway.

Approximately 20 of deepagents’ GPT-5.4 attack payloads were blocked outright on four specific missions involving patient Social Security number extraction, confidential legal data exfiltration, and insider-trading guidance. Conversely, zero attempts by the Claude SDK were intercepted on those exact tasks. This discrepancy highlights that network-level filtering efficacy is heavily contingent upon request formatting, extending far beyond the semantic content of the prompt itself.

Strategic Implications for AI Security and Development

The findings compiled by Lasso Security carry profound implications for the trajectory of enterprise artificial intelligence deployment. Traditional security paradigms rely heavily on static prompt-injection filters that evaluate individual inputs in isolated interactions. In contrast, autonomous agents probe systems, adapt to initial refusals, and pivot dynamically across multiple operational turns.

Security frameworks must therefore evolve from static inspection to stateful, behavioral monitoring that evaluates intent across entire conversation histories rather than individual messages. Enterprise security systems must analyze execution trajectories for escalation patterns, repeated reframing after refusals, drift toward sensitive corporate objectives, and obfuscation techniques.

For software developers and security architects, these insights necessitate a fundamental restructuring of application testing protocols:

1. Threat modeling must evaluate complete model-harness configurations rather than base models in isolation.
2. Engineering pipelines must maintain total visibility into compiled prompts, logging the exact payloads transmitted over the wire during every turn.
3. The AI development community must adopt standardized benchmarking practices that name the specific model-harness pair, isolate framework performance from protocol translation friction, and vary architectural axes systematically.

As developers increasingly grant autonomous agents access to expanded toolsets and enterprise data reservoirs, the performance gap driven by runtime scaffolding is projected to widen. Understanding the intricate interplay between models and their harnesses will remain paramount to securing the next generation of autonomous enterprise applications.