NCC Group’s AI Red Team recently published key findings after penetration testing dozens of AI applications. Analyzing Secure AI Architectures reveals that major AI vulnerabilities stem not from model flaws, but from misunderstanding how AI systems interact with traditional application components.
Rather than “patching” AI models with guardrails against prompt injection, organizations should implement architectural patterns that limit attack impact through proper trust segmentation. That’s because no reliable way exists to expose an AI model to untrusted data and simultaneously trust its output.
The Security Paradigm Shift
Even organizations with theoretically secure AI models remain vulnerable when attackers exploit weaknesses in architectural design and trust relationships. NCC Group’s blog highlights two fundamental shifts that security and development teams must understand:
- Code-centric → Data-centric: Security now requires controlling how data flows through systems and establishing trust boundaries, similar to IAM’s focus on access control.
- Reactive patches → Preventive architecture: Resources spent on prompt injection guardrails would be better invested in architectural solutions applying IAM principles of defense in depth.
Secure AI Application Architecture Patterns
NCC Group has identified several architectural approaches that have proven effective in real-world scenarios:
- Gatekeeper Pattern: Separate models with different trust levels and capabilities. User-facing models handle sensitive functions, while data-facing models process untrusted inputs but lack privileged access. Implements IAM’s separation of duties.
- Orchestration Tree Pattern: A trusted coordinator model assigns tasks but never processes untrusted content. Information flows downward to specialized models with minimal permissions. Embodies IAM’s least privilege principle.
- State Machine Pattern: Permissions shift based on current needs, triggered by explicit user actions rather than potentially manipulated data. Implements IAM’s just-in-time access principles.
Consider integrating these patterns into your reference architectures.
Practical Implementation Advice
NCC Group’s research also provides valuable guidance on implementing these secure AI architectural patterns spanning:
- Data type conversion across trust zones
- Content tagging & masking
- Trust-level separation
- Trust boundary testing
Check out the full blog to explore these implementation strategies in depth.
IAM Fun Fact
In AWS, attaching a Lambda function running an LLM to an over-permissioned role can allow prompt-injected outputs to trigger actions like deleting S3 buckets, if model outputs aren’t gated by explicit authorization logic.
💡 NCC Group’s architecture patterns act like IAM at the system design level enforcing trust boundaries that IAM policies alone can’t.
Sources:
- AWS IAM Best Practices – Least Privilege: https://aws.amazon.com/iam/resources/best-practices/
- AWS Lambda Access Control: https://docs.aws.amazon.com/lambda/latest/dg/access-control-overview.html
Additional Resources
For teams looking to implement these approaches, we recommend:
- Analyzing Secure AI Architectures – Full blog by David Brauchler III
- OWASP LLM Top 10 – Complementary security guidance
- Microsoft AI Adoption Framework – Additional architectural perspectives
- 12-Factor Agents – Principles for building reliable LLM applications
👏 We extend special recognition to NCC Group’s AI Red Team for developing practical, field-tested architectural patterns that fundamentally enhance AI security posture by applying classic IAM principles to the unique challenges of AI systems. Their approach provides a clear path forward for organizations building trustworthy AI applications.
Stay up to date with The Effective IAM Newsletter:
Recent Comments