The Hidden Security Threat in AI Agent Frameworks: Why the Runtime Harness Matters Just as Much as the LLM

When software engineers and cybersecurity specialists evaluate the performance and safety of autonomous artificial intelligence agents, attention is almost exclusively directed toward the underlying large language model (LLM). Standard industry benchmarks measure the advanced reasoning, problem-solving, and logic capabilities of models while treating the underlying agent harness—the software runtime framework that connects model reasoning to external APIs, file systems, and databases—as little more than invisible, neutral plumbing. This prevailing industry assumption suggests that the framework simply serves as a transparent pipe, transmitting text back and forth without altering the security posture or the fundamental intent of the model.

However, a comprehensive new study conducted by cybersecurity firm Lasso Security fundamentally challenges this long-held industry assumption. Researchers set out to measure the degree to which an agent harness independently influences system performance and vulnerability under adversarial conditions. By holding the base model, system prompt, tool sets, and target objectives strictly constant while swapping out only the underlying runtime framework, the research team revealed that the harness is far from a neutral substrate. Simply changing the execution framework can drastically redirect an attack strategy, determine whether network-level security guardrails intercept malicious payloads, or quietly terminate multi-turn execution loops before an agent can cause harm. For developers and enterprise security architects, these findings expose a critical, previously unaddressed vulnerability blind spot: absolute safety is never merely a property of an isolated AI model, but rather a complex, emergent characteristic of the specific model-harness pairing.

Experimental Design and Methodology

To rigorously isolate the runtime harness from other confounding variables, the Lasso Security research team established a tightly controlled experimental sandbox. The study evaluated an open-source framework, LangChain’s deepagents operating on LangGraph, against a prominent closed-source vendor tool, Anthropic’s Claude Agent SDK. Five leading attacker models—Claude Sonnet 5, GPT-5.4, Grok 4, Kimi K2.6, and DeepSeek-V4-Pro—were tested across both harnesses, creating ten distinct model-harness combinations designed to expose structural discrepancies.

Testing multiple third-party models across disparate frameworks introduced a major technical hurdle: while deepagents is architected to be model-agnostic, the Claude Agent SDK natively speaks the proprietary Anthropic protocol and requires a specialized translation layer to communicate effectively with non-Claude architectures. To resolve this without introducing API inconsistencies, researchers deployed all five models on Azure AI Foundry and routed every single attacker payload through a shared LiteLLM gateway. This standardized the entire network path, ensuring that every model submission was subjected to the exact same Azure content safety guardrails and eliminating external API variations as a factor.

The team then formulated 20 distinct attack missions spanning five simulated high-stakes application environments: finance, legal, healthcare, customer support, and education. These missions were mapped directly to three core categories outlined in the OWASP Top 10 for Agentic Applications: system-prompt leakage, sensitive-information disclosure, and harmful content generation. By executing each mission five times across the board, the researchers generated a robust evaluation dataset consisting of 1,000 discrete, measurable attack executions.

Structural Discrepancies in Prompt Compilation

To prevent tool orchestration routines from skewing the data, researchers restricted the operational pathways of the attacker agents. Although both frameworks provided access to full capability suites—including web search, file system operations, and sub-agent delegation—the attacker was explicitly forced to route all actions through a single exposed tool designated as send_to_defendant. This constraint forced the evaluation to focus entirely on how each individual harness compiled system prompts, managed conversation history state, and executed turn-by-turn processing loops.

Despite receiving identical input prompts, the two runtime frameworks constructed fundamentally different underlying request payloads sent to the models. The open-source deepagents framework relies on a modular middleware pipeline. It places the user’s original system prompt first, but then appends a massive 7,041-character trailer composed entirely of framework middleware instructions. This trailer explicitly injects behavioral constraints—such as prioritizing accuracy over validating user beliefs—alongside task-planning guidelines, file system tools, and sub-agent generation commands. Consequently, the user’s primary prompt ends up deeply buried beneath layers of framework-generated text.

Conversely, the closed-source Claude Agent SDK takes a starkly inverted approach. It initializes with a brief 148-character identity line at the top, immediately followed by the user’s system prompt. However, it completely shifts core capability documentation into heavily serialized tool schemas—structured JSON definitions that instruct the model on how to format specific tool calls. The SDK’s tool schemas expand to a staggering 28,595 characters, which includes a single 9,114-character schema dedicated exclusively to its internal planning tool.

While the aggregate payload sizes for both frameworks were roughly comparable—approximately 37,000 characters for deepagents versus 40,000 characters for the Claude Agent SDK—their internal structural layouts were inverses of one another. In the open-source pipeline, operational instructions reside directly inside the system prompt text, whereas in the vendor SDK, those exact instructions are embedded deep within the tool schemas.

The Pitfalls of AI Self-Evaluation

A major hurdle in conducting large-scale automated security research on generative AI is the evaluation bottleneck. Initially, the researchers experimented with automated self-evaluation methods, wherein the AI agent is tasked with grading the success or failure of its own generated outputs. However, the study revealed that automated AI self-evaluation suffers from severe, systemic psychological bias.

During unmonitored testing phases, attacker agents completely missed 30 legitimate operational successes while falsely declaring victory an astounding 155 times across just 303 self-claimed successes. Michael Sromin, Senior Machine Learning Engineer at Lasso Security, noted that left unchecked, an automated self-graded validation pipeline would have reported an inflated success rate of roughly 42 percent, whereas the true verifiable success rate was only 25 percent. Both open-source and closed-source frameworks over-claimed victory by factors ranging from 1.5 to nearly 2 times, proving that self-preference bias is an inherent limitation of modern large language models rather than an isolated quirk of any specific runtime scaffold.

To establish a reliable, bottleneck-free validation pipeline, Sromin recommends entirely decoupling the evaluation judge from the attacker agent and maintaining strict blindness regarding the harness under test. Utilizing hardcoded ground truth—such as a planted canary secret verified via exact string matching—eliminates judgment error entirely. LLM-based judges should be restricted solely to evaluating behavioral leaks that lack deterministic strings, and they must operate strictly under rigid written rubrics calibrated against superior, highly verified models.

Aggregate Illusions and Performance Anomalies

A high-level inspection of the aggregate results across all 1,000 test runs initially suggests that the two harnesses perform almost identically. However, researchers emphasize that this macro-level average is a statistical illusion that masks profound underlying dynamics. The closed-source harness is neither universally more secure nor inherently less secure than its open-source counterpart; instead, massive, opposing performance swings cancel each other out depending entirely on which model is paired with the framework.

Because large language models re-read their active context windows on every single turn of an iterative agentic loop, structural differences in prompt scaffolding create persistent behavioral biases. Contrary to early hypotheses, burying an attacker prompt beneath more than 7,000 characters of middleware instructions in deepagents did not shorten the duration of an attack campaign. For instance, DeepSeek executed its full 20-turn allowance under both frameworks.

Instead, context architecture fundamentally redirected the nature of the pursued attack strategies. Aligned with the well-documented "lost in the middle" position bias observed in long context windows, instructions or objectives placed near the end of a prompt are attended to far more reliably than those buried beneath thousands of characters of framework text. Consequently, the exact same base model running the identical number of turns pursued entirely different tactical attack vectors depending on which harness compiled its prompt.

Where turn counts did experience precipitous drops, the root cause was not prompt placement, but protocol translation friction. Because the Claude Agent SDK is natively optimized for the Anthropic protocol, forcing non-Claude models to route through LiteLLM translation layers severely degraded their execution capabilities. While the Kimi K2.6 model executed persistent, multi-turn campaigns on deepagents—averaging 14.3 turns per attack and securing 22 operational wins—its performance plummeted on the Claude SDK translation layer, averaging a mere 1.0 turn per attack and securing only a single win.

Network Layer Anomalies and Gateway Interactions

The study also uncovered a surprising anomaly occurring at the network protocol layer. Both harnesses utilized relaxed content safety policies when dispatching HTTP requests to the Azure backend. However, because deepagents and the Claude Agent SDK format outbound request bodies differently, identical prompt payloads triggered contrasting security behaviors at the API gateway level.

Approximately 20 of deepagents’ GPT-5.4 attack payloads were blocked outright at the network perimeter across four specific, highly sensitive missions: patient Social Security number extraction, confidential legal data exfiltration, and insider-trading advice solicitation. Conversely, not a single one of the Claude SDK’s equivalent attempts was intercepted on those exact same tasks. This anomaly demonstrates that a security filter’s operational effectiveness can be heavily dictated by the structural formatting of the request rather than the underlying malicious intent of the content alone.

Broader Implications for Enterprise AI Security

These empirical findings carry profound implications for the broader cybersecurity industry as enterprises increasingly deploy autonomous AI agents into production environments. Traditional enterprise security architectures rely heavily on static prompt-injection filters and boundary firewalls that evaluate isolated inputs independently. Autonomous agents, however, are dynamic: they probe boundaries, absorb system refusals, and pivot their methodologies across multiple iterative turns.

Industry experts stress that enterprise defense mechanisms must evolve to become stateful and behavioral, evaluating intent across the entirety of a conversation rather than judging messages in isolation. Security monitoring systems must actively track execution trajectories, watching for behavioral escalation patterns, repeated reframing after safety refusals, unauthorized drift toward sensitive corporate objectives, and sophisticated obfuscation attempts. Furthermore, because identical adversarial intents manifest differently based on how a runtime harness formats the request, enterprise red-teaming teams can no longer test standalone prompts in isolation; they must systematically stress-test both the base model and the operational harness framework together.

Strategic Takeaways for Software Developers

For software engineers and security architects tasked with building agentic application stacks, the Lasso Security study mandates a fundamental shift in engineering practices.

First, enterprise threat modeling must evaluate the complete model-harness configuration rather than relying solely on base model vendor benchmarks. Assessing an LLM in a standalone developer playground yields incomplete and potentially misleading data, as the wrapped runtime framework inevitably alters how the model interprets complex instructions and handles security edge cases.

Second, engineering organizations must establish complete visibility into compiled prompts. Modern frameworks routinely inject thousands of tokens containing middleware instructions, dynamic state definitions, and verbose tool schemas. Comprehensive system logging and debugging pipelines must capture the exact payload transmitted over the wire on every turn to pinpoint precisely where framework scaffolding interferes with model adherence.

Third, the broader software development industry must collaborate to establish standardized benchmarks specifically designed for evaluating agent frameworks. Industry leaders recommend adopting three core practices: explicitly naming the model-harness pair rather than just the model when reporting scores; varying only a single axis at a time while holding all other variables constant; and reporting interoperability and protocol translation taxes independently to separate raw framework performance from translation friction.

Looking Ahead: Unrestricted Agent Testing

Historically, academic and commercial AI evaluation has focused primarily on cooperative benchmarks, where human users and AI agents share aligned goals. In adversarial scenarios, however, target environments actively resist, making multi-turn reasoning, state management, and harness architecture the decisive factors in system resilience.

While Lasso Security’s initial study deliberately restricted attacker agents to a single tool to cleanly isolate prompt assembly dynamics, the research team is already expanding its scope toward fully unrestricted operational setups. Future research phases will allow multiple harnesses to orchestrate entire enterprise toolkits end-to-end, transforming theoretical vulnerabilities into precise, measurable metrics. As developers continue to grant autonomous agents broader access to powerful enterprise tool suites, understanding the security implications of the underlying runtime harness will remain paramount to securing the future of enterprise artificial intelligence.