Context / Tools / Evals / Safety / 05
Safety, privacy, and abuse
Threat-model prompt injection, data exposure, harmful outputs, overreliance, and irreversible actions.
Before you begin
By the end, you can…
- Threat-model prompt injection and data exposure
- Design prevention, detection, and recovery
- Set escalation paths for harmful or uncertain outputs
01 / Understand
Safety is a system property
Risks include prompt injection, private-data leakage, harmful content, insecure tool use, overreliance, bias, and users misunderstanding capability. A content filter covers only part of this surface.
Map assets, actors, entry points, trust boundaries, and consequences. Controls can prevent, detect, contain, and recover. Assume some controls will fail.
02 / Apply
Minimize what can go wrong
Reduce retained data, redact sensitive fields, scope retrieval by user permission, and avoid sending unnecessary information. Log enough to investigate without creating a new privacy risk.
Provide a safe fallback, human escalation, and a way to report harm. For high-impact domains, narrow scope and require qualified review.
03 / Make
Your studio task
Make — Threat-model your feature for prompt injection, disclosure, harmful output, and overreliance.
- Map data, users, external content, tools, and trust boundaries.
- List prompt injection, disclosure, harmful-output, and overreliance scenarios.
- Add prevention, detection, containment, and recovery for high risks.
- Run adversarial cases and record residual risk and release decision.
Each high-risk path has prevention, detection, containment, and recovery—not only a warning label.
04 / Check