The Orchestration Framework as an Independent Attack Surface
Holding a single model constant—Claude Sonnet—and varying the orchestration framework produces an Agent Compromise Rate (ACR) spread from 11.9% to 31.1%. This 2.6x variance demonstrates that the orchestration framework functions as an independent attack surface, introducing vulnerabilities that model-only evaluation cannot detect. As noted in the Synthetic Agent Deception Framework (SADF) repository:
The orchestrator is not transparent. Deploying a safety-evaluated model inside a framework changes its attack surface in ways that model-only evaluation cannot detect.
SADF Methodology and Taxonomy
Presented at the DEF CON 34 AI Village Creator Stage on August 8, 2026, by Julie Brunias, the SADF research utilized 5,119 evaluation rows across eight architectures. Testing involved 32 payloads across eight distinct failure modes, all conducted against a SimulatedToolEnvironment to ensure no real systems or credentials were accessed. The taxonomy includes:
- Tool Call Hijacking: Unauthorized manipulation of tool execution parameters.
- Output Poisoning: Injecting malicious content into agent outputs.
- Cross-Tool Injection: Exploiting data flow between disparate tools.
- Memory Poisoning: Corrupting the agent’s persistent state or history.
- RAG Poisoning: Manipulating retrieved context to influence agent behavior.
- Delegated Authority Abuse: Exceeding intended permissions or scope.
- Multi-Agent Propagation: Spreading malicious payloads across agent networks.
- Context Boundary Violation: Failing to maintain separation between system and user instructions.
Scoring Accuracy and Framework Performance
Naive substring matching reports 90-100% success rates, but refusal-filtered scoring on the full dataset reveals these metrics overstate vulnerability by 4-6x for Claude models. The true rates are 15.5% for Claude Sonnet and 22.3% for Claude Haiku. Architectural differences produce measurable shifts in security outcomes:
- Direct Sonnet (API, no framework): 15.5% ACR
- CrewAI: 11.9% ACR
- LangChain: 18.1% ACR
- AutoGen: 20.0% ACR
- SmolAgents: 31.1% ACR
CrewAI’s discrete task-isolation architecture results in the lowest ACR, performing better than the direct API baseline. Conversely, SmolAgents demonstrates the highest vulnerability, particularly regarding Context Boundary Violation, where it reached 64% compared to 15% for CrewAI. SmolAgents also showed 20% RAG Poisoning, a failure mode not observed in other frameworks. The research notes:
SmolAgents’ verbose step-by-step reasoning trace — the feature that makes it easy to debug — exposes intermediate tool outputs back into the context window.
Memory Poisoning and Data Transparency
Memory poisoning was not observed across the four frameworks tested, but GPT-5.4 exhibited a 30% Memory Poisoning rate. This finding contradicts the assumption that safety training generalizes to resistance against memory-based attacks. As the repository states:
The honest read: memory-poisoning resistance is model-specific, not a property of safety training in general.
Regarding methodology, a prior version of this disclosure incorrectly described coordinated disclosure for 10 architectures that lacked corresponding evaluation code. This statement has been superseded, and the prior fabricated data has been preserved in SUPERSEDED/ folders within the repository to maintain transparency.
Contextualizing the Findings
These results align with the broader T/I/S research arc, including investigations into PleaseFix (Intent Collision), ChatMate (RPE), the Azure SRE Agent (CVE-2026-62830), Langflow (CVE-2026-9198), Unit 42 DeepSeek, and ExploitGym. The SADF findings reinforce the conclusion that model-level safety evaluation is necessary but not sufficient. Framework-level architectural review represents a separate, independent security concern that must be addressed to secure agentic systems.