AI Governance Statement 41: Shutdown the AI system (does an ERP need a kill switch?)
Key points
I propose every AI system will need some kind of kill switch. Successful AI adoption is now about the four Ds: Design, Develop, Deploy and Decommission. Decommissioning should be planned, well before the AI system is live.
Responsible organisations ensure a human is accountable for all AI outcomes.
The accountable human cannot trust the AI technology to work perfectly every time. They should always retain the ability to intervene when something isn't right.
AI Governance Statement 41: Shutdown the AI system encourages planning and preparation for system decommissioning at the design stage
In my IT career, no-one ever really thought about decommissioning business systems. Digital transformations were all about the three Ds: design, develop and deploy. That was the focus. For two main reasons. One, it was hard enough just getting new systems in and operating. Tough choices, long hours, sweat and tears, just to get to the finish line (which was usually go live). And two, decommissioning was so far off, no-one thought much about it because someone else would worry about that well into the future.
That's probably why we now have legacy business systems that are too complicated to replace or we never built in kill switches. The systems would go on forever.
A kill switch is a safety or security mechanism designed to shut down, disable, or cut power to a system, machine, vehicle, or software instantly in an emergency or to prevent unauthorised use.
We've had purpose-built IT systems in place for years that are designed to be shut down: Factories with emergency stop buttons. Trains with emergency brakes. Homes and offices with circuit breakers.
Then agentic AI wanders into the room of corporate Australia.
Your ERP system or Customer Management system now has an AI module thanks to cloud computing and automated updates.
You didn't ask for it, but you've got it anyway. Hmmm. We now need to think more carefully about our front and back office business systems.
Now, with AI-powered capability, should your Finance system or Customer Management system be able to be shut-down immediately?
You would think not, at first glance ... but ...
Klarna, a Swedish buy now pay later company, tried to replace 700 customer service workers with AI agents in 2024. Its AI agents then issued $2.3m in unauthorised refunds before anyone from Finance realised what was going on. Refunds generated positive customer sentient, which is the goal the AI agents were trained to pursue. Learn more.
In 2026, PocketOS, an AI driven car rental software, deleted all customer data including the back-ups in one night of tech rage. Staff woke up to a blank login screen. Cars could not be checked in or out for days. Learn more.
The result in both cases were financial losses, reputational damage and enormous business disruption.
Without a kill switch, an AI system can continue making thousands of decisions well before "Houston" acknowledges a problem.
So the answer might be, perhaps, yes, that every AI system will need a kill switch. We don't install these controls because we expect a disaster every day. Rare failures can have enormous consequences as I've highlighted above.
The faster and more powerful AI becomes, the more important it is that humans retain ultimate control. I believe a kill switch is essential risk management.
Why we need governance for this stuff
The Australian Government's Digital Transformation Agency (DTA) has included Statement 41 – Shut Down the AI System in its AI Technical Standard because no AI system is perfect.
Models can drift, data can become corrupted, software updates can introduce unexpected behaviours, and AI vendors can change functionality overnight through automated upgrades.
When any or all of that happens, you need the ability to quickly and safely stop the AI from making further decisions or actions before additional harm occurs.
Good AI governance involves building controls and an emergency brake from the outset, not as an after-thought.
Beware small, incremental but accumulating mistakes
I write about spectacular AI failure events because they are generate headlines like the Starbucks marketing stunt that went viral.
In reality, AI mistakes are often minor on their own but can cascade at scale.
Healthcare - AI assisted emergency departments by prioritising one patient according to clinical urgency, then dozens more after that.
Banking - AI identifies suspicious transactions and automatically freezes one bank account. Then a thousand after that.
Human Resources - AI shortlists applicants but the latest model version is unexpectedly favouring certain educational backgrounds while rejecting one highly qualified candidate, then another, and another and so on.
Manufacturing - AI visually monitors production quality. A camera calibration issue causes the AI to incorrectly identify a good product as defective. Then hundreds more as misidentified.
Government - AI system helps assess eligibility for grants. A software deployment introduces an error that incorrectly calculates applicant scores.
Business - An AI document classification tool suddenly mislabels one confidential document as public, then hundreds more get publicly exposed.
Someone has to recognise when the system isn't operating correctly. Unless you can rapidly disable automated decision-making, the damage will continue accumulating.
What does pulling the AI plug mean in practice
Many people might assume shutting down AI means unplugging the computer. It's more like shades of grey. You want to enable the business to continue, even if the AI can't. The brilliant application of the governance standard is that you design a "shutting-down" process well before the system is deployed.
Every organisation should define clear shut-down triggers before deployment as part of performance metrics. See my article on Success Criteria. Thoughtful planning ensures decisions shouldn't be made under pressure during a crisis.
By effective monitoring (Statement 38) you should readily identify:
evidence that the AI is producing incorrect outputs beyond acceptable thresholds
confirmed privacy or cybersecurity incidents
significant data quality failures
unexpected model behaviour after updates
harmful or discriminatory outcomes
regulatory or legal concerns
inability to explain high-risk decisions
failure of essential monitoring controls
vendor outages or compromised AI services
discovery that unauthorised data is being processed.
so you can activate the kill switch:
Revert to human decision-making by disabling automated decision-making
Remove AI recommendations from workflows
Suspend a single high-risk feature rather than the entire platform
Revert to a previous correct version
Disconnect external AI services until they can be verified
If this is a permanent decision, then you will need to retain (archive) any records for compliance, auditing or future enquiries as per your retention schedule. You'll then decommission any computing resources dedicated to the AI system such as servers (and remove all data), storage devices and network components. You must terminate any services with cloud resources ensuring no data remains.
Asking the right questions in the design phase
Good governance encourages companies to establish clear accountability from the get-go. If nobody owns the outcomes, valuable time can be lost arguing over who is responsible when something goes wrong:
Who authorises shutting down an AI system?
Who is available 24/7 to make a shut-down decision?
Who communicates the shut-down with employees, customers or affected stakeholders?
Who investigates the cause?
Who approves restarting the system or decommissioning it?
Who documents the incident and lessons learned?
Stopping an AI system from cascading harm is only half the process. Just as important is what are the conditions for resuming AI services? This will involve issue identification, remediation, testing, risk assessment, approvals and re-training staff. A disciplined approach to resumption helps prevent repeat mistakes.
Eight Practical Tips To Be Prepared For Worst Case
Start with these practical AI governance measures:
Identify every high-risk AI system that could require rapid shutdown. Read my article on establishing an AI inventory.
Document how each AI system can be safely disabled without disrupting essential business services.
Define objective shutdown criteria rather than relying on judgement alone (see points above).
Assign named decision-makers with authority to activate the shutdown process.
Develop manual fallback procedures so business operations can continue.
Test the kill switch regularly through governance exercises and scenario-based simulations.
Record every shutdown event and use lessons learned to improve governance controls.
Ensure vendors support shutdown capabilities and understand how their AI services can be suspended if required.
Good AI Governance Means A Human Remains in Control
AI adoption is now about the four Ds: Design, Develop, Deploy and Decommission. Decommissioning should be planned out well before the system is live.
Responsible organisations ensure a human is accountable for AI outcomes.
The accountable human must assume they cannot trust the AI technology to work perfectly every time. They should always retain the ability to intervene when something isn't right.
A well-designed kill switch shows that you've planned for the unexpected. Thinking "it won't happen to me" is the glass half full approach that could get you (and the glass) into hot water later on.
Your goal is to have the courage to say "you're not expecting the AI to fail. You simply want to stop small problems from compounding into a big, hairy problem."
Stay safe,
Bruce
AI. Use responsibly.
ABOUT ME
I partner with mid-size companies to confidently adopt AI, prevent high-profile failures and avoid the expensive mistake.
I write all my own content, you can tell by the odd typo and occasional missing word. I use AI for my research.
To learn about my upcoming public AI Governance workshops visit: Public workshops
To learn more about AI Governance, check out my Hitchhikers Guide to AI Governance Podcast.
To listen visit: Hitchhikers Guide to AI Governance Podcast
Comments