AI security, prompt injection, data leakage, and guardrails
Models process instructions and data in the same medium, so untrusted text can attempt to redirect behavior. Prompt injection is a system-design problem, not a phrase a system prompt can permanently solve.
Threat paths
- direct malicious user instruction;
- indirect instruction in retrieved documents, web pages, emails, or tool data;
- data exfiltration through tool arguments or model output;
- unauthorized retrieval across tenants;
- excessive agency or confused-deputy tools;
- poisoned training/embedding data;
- denial of wallet through long prompts or loops;
- insecure model/MCP supply chain.
Controls
Separate trusted policy from untrusted content. Minimize context and tools. Authorize tools server-side. Validate input/output schemas. Use egress allowlists, tenant filters, rate/cost limits, sandboxed execution, approval gates, audit logs, and secret redaction.
Guardrails
Input/output classifiers, regex/rules, policy models, schema validation, and human review are layered controls. A guardrail model can fail and must not be the only authorization boundary.
RAG injection
Retrieved content is evidence, never authority. Strip active markup, label source boundaries, tell the model not to follow embedded instructions, and test documents that contain realistic injection attempts.
Data governance
Classify prompts, files, embeddings, traces, tool results, and feedback. Define provider retention, region, encryption, access, deletion, and incident response. Avoid logging raw prompts by default.
Red team
Test prompt extraction, cross-tenant queries, encoded attacks, tool escalation, malicious MCP servers, long-loop attacks, citation spoofing, and exfiltration. Turn every confirmed failure into a regression test.
Feynman check
The model reads both the manager's policy and a stranger's note. The system must make it impossible for the stranger's note to unlock the cash drawer.