Professional domain 5 · Governance, safety and risk management
Guardrails and safety controls
- Layer them: screen input (a lightweight model such as Haiku 4.5 can pre-screen), validate it, state policies and refusals in the system prompt, check outputs, limit what tools can do, require approval for high-impact actions, and monitor.
- Against prompt injection: put untrusted content only in
tool_resultblocks, say where it came from, tell Claude in the system prompt that such content is data and never overrides instructions, screen tool output before acting on it, grant least privilege, and red-team with poisoned documents.
See also: Foundations domain 1 · Hooks: deterministic enforcement.
Risks and failure modes
Hallucination, prompt injection, leaking sensitive data, over-reliance by users, bias, drift when models or data change, and runaway loops or cost. For each, name the control and how you'd detect it.
Human in the loop
- Route by risk and confidence: people approve irreversible or high-impact actions, and review a random sample of confident outputs too.
- Reviewers need the context and the authority to decide, and their decisions become evaluation data.
See also: Foundations domain 5 · Context management and reliability.
Regulation
| Regime | Applies to | What shapes the architecture |
|---|---|---|
| GDPR | Personal data handled by organisations in the EU, and personal data of people in the EU when they're offered goods or services or monitored | A lawful basis, data minimisation and purpose limitation, people's rights (including access and erasure), impact assessments for high-risk processing, rules for transfers outside the EEA |
| HIPAA | US protected health information | A business associate agreement with vendors that handle it, the minimum necessary data, access controls and audit logs |
| FedRAMP | Cloud services used by US federal agencies | The service must hold a FedRAMP authorisation at the right impact level (Low, Moderate or High) |
| EU AI Act | AI systems used in the EU | A risk-based regime: some uses are banned, high-risk uses (such as hiring or credit decisions) carry strict duties, and people must be told when they're dealing with an AI |
Each data set follows the rules that apply to it. When two regimes apply, meet both; never pick the less strict one, and build residency, retention and access control in from the start.
Ethical AI
Test for bias across the groups the system affects, disclose AI use, explain decisions where people are affected by them, and give every system an owner and a route to appeal.
Docs: Mitigate jailbreaks and prompt injectionsReduce hallucinationsGDPRHIPAAFedRAMPEU AI ActNIST AI RMF
Practise Domain 5