AI Governance Statement 11: Design Safety Systemically
We often associate safety with physical harm like falls, cuts, abrasions and fractures. If you walk onto any building site in Australia, you will see safety everywhere. Signage, hi-vis vests, areas cordoned off, scaffolding for working at heights and so on. It wasn't always this way, but you can actually see, feel and touch safety. Carpenters using circular saws, welders welding, painters using chemicals, roof plumbers laying tiles with safety in mind. A typical safety mantra is to ensure construction workers clock-off unharmed and return to their families at night.
The trouble with "safe" computer systems is they exist in a virtual world. In the physical world, safety professionals talk about hazard identification, incidents, near misses and safety is everyone's job. Things we can see and conceptualise. How well does this translate to AI? What harm could a seemingly benign computer system do? And why do we need to be concerned about digital safety?
Thankfully, my dutiful digital assistant, Chatgpt, kindly provided a list of more than 150 harmful AI scenarios. Choosing one harm per category, you can easily see scale and breadth of AI's reach to do non-physical harm.
Safety cannot be added later
One of the most significant safety risks arise when several small failures compound to cause cascading harm.
A newly retired couple seek financial planning retirement advice.
The staff use AI to generate advice and financial plans. The AI hallucinated some of the information.
The staff member assumed the output was correct.
There is no human review.
The incorrect information is provided to the retirees in a credible, detailed, long-form financial plan.
The retirees rely on the information as provided and subsequently suffer a financial loss.
The financial planning / accounting practice faces complaints, legal action and reputational damage.
This failure wasn't isolated just to one client. The same mistake was repeated over and over again, cascading the harm across many clients. It starts small, but if undetected, the small mistakes compound over time.
Ernest Hemingway once wrote about going bankrupt: "Two ways. Gradually, then suddenly."
Every AI use case resulting in harm follows that same three step pattern:
Most systems follow the same path: Build the system first, test it later, fix any problems after deployment. Unfortunately, that approach works not-so-well when it comes to safety. Once an AI system has been trained, integrated into business processes and relied upon by staff or consumers, redesigning it to become safe is often expensive, time-consuming, disruptive and worst case, have to start all over again.
That is exactly why the Australian Government's Digital Transformation Agency (DTA) Technical Standard for AI includes Statement 11 – Design Safety Systemically. Rather than treating safety as a final checkpoint, Statement 11 requires companies to embed safety throughout the entire AI lifecycle: from planning and design through development, testing, deployment, monitoring and retirement.
Safety is not a feature to be added as a module. Safety is something to be architected in every part of an IT system.
Safety by design
Systemic safety means looking well past accurate outputs. It requires an understanding of how the AI system interacts with people, business processes, technology and organisational decision-making.
Many AI failures occur not because the model itself is defective, but because the surrounding system was poorly designed. One of the most practical ways to implement Statement 11 is to think in terms of layered safety controls. Rather than relying on a single safeguard, companies should build multiple independent protections that reduce the likelihood of harm.
For example, an AI system generating correspondence for consumers could include:
input validation to prevent inappropriate requests
guardrails restricting sensitive outputs
mandatory human approval before external release
audit logging of every AI interaction
ongoing monitoring for unusual behaviour
incident reporting and
rapid rollback capability or kill switch
If one control fails, the remaining controls continue protecting from harm.
This "defence in depth" approach is already common practice in cyber security and aviation.
A defense-in-depth approach is a cybersecurity strategy that uses multiple overlapping and redundant security layers—such as physical, technical, and administrative controls. If one barrier fails, subsequent layers still protect the system.AI governance applies the same philosophy.
Ethics Principles are a useful guide
If it doesn't feel right. It probably isn't. Australia has developed its own Artificial Intelligence (AI) Ethics Principles to guide businesses and governments to responsibly design, develop and implement AI. The voluntary principles help:
achieve safer, more reliable and fairer outcomes
reduce the risk of negative impact on those affected by AI applications
businesses and governments to practice the highest ethical standards when designing, developing and implementing AI.
Free Assessment Tools (Safety by Design)
Developed in Victoria, Alt-TAB, is a comprehensive, free, pre-deployment ethics and Safety by Design assessment for AI and emerging technology. The assessment tools can be used by companies or software vendors to identify harm pathways before they deploy AI.
The assessment provides a helpful readiness score and breaks down blind spots and risks across eight relevant categories. Here's one clip from an assessment report.
Here's a link to the assessment tool: https://alttab.afk.org.au/
The charity that designed the tool, Away From Keyboard, supports children, carers, women, and vulnerable communities across Australia and contributes to international conversations on ethical AI, child safeguarding, and technology-facilitated gender-based violence prevention.
Practical AI Governance Safety Implementation Tips
Workshop it.
Start by conducting structured "hazard" workshops in the design stages, well before development begins. Bring together managers, technical specialists, legal advisers, privacy experts and frontline staff to identify how the AI could fail and who could be harmed.
Map it.
Identify every point where the AI influences decisions. Even if the AI only provides recommendations, understand how those recommendations affect downstream business processes.
Mitigate it.
Introduce multiple "fail-safe" check-points. If confidence scores are low, data quality is poor or unusual requests are detected, the AI should escalate to a human in the loop rather than continue generating potentially unsafe outputs.
Test it.
Test for unintended consequences (Statement 30) rather than your common or typical scenarios. Many AI incidents occur because of unusual situations that were never considered when the AI was envisioned.
Monitor it.
Finally, establish continuous monitoring after deployment. AI systems change over time as user behaviour, data quality and operating environments evolve. Safety therefore becomes an ongoing operational responsibility rather than a one-off project outcome.
Safety is everyone's responsibility
The most important message from Statement 11 is that AI safety is not solely the responsibility of IT or compliance.
Managers determine how AI is used. Procurement teams influence vendor requirements. Executives establish risk appetite. Governance committees provide oversight. End users make decisions based on AI recommendations.
Designing safety systemically means recognising that every part of the company contributes to safe AI outcomes.
By embedding safety into the architecture of AI solutions, you can confidently harness the benefits of AI while protecting consumers, employees and public trust. Now, where did I leave my hi-vis vest.
Stay safe,
Bruce
AI. Use responsibly.
ABOUT ME
I partner with mid-size companies to confidently adopt AI, prevent high-profile failures and avoid the expensive mistake.
I write all my own content, you can tell by the odd typo and occasional missing word. I use AI for my research.
To learn about my upcoming public AI Governance workshops visit: Public workshops
To learn more about AI Governance, check out my Hitchhikers Guide to AI Governance Podcast visit: Hitchhikers Guide to AI Governance Podcast
Comments