Artificial Intelligence (AI) is a genuine productivity multiplier, and Australian workplaces have embraced it fast. According to the 2025 Cost of a Data Breach Report, shadow AI has raised the average cost of data breaches, and more incidents now involve the compromise of personal identifiable information (65%) and intellectual property (40%) with much of this happening through personal, unmanaged accounts, invisible to the organisation. One of the emerging risks facing organisations and governments is the inadvertent disclosure of sensitive information through AI platform usage. Well-meaning staff who adopt AI tools to improve their productivity and efficiency often do so without recognising that the information they input maybe transmitted to, retained by, or used to train third-party AI models without their knowledge or explicit consent.
For ease of use, in December 2023, a child protection worker in Victoria used ChatGPT when drafting a Protection Application Report in putting significant amount personal and delicate information. This resulted in a privacy breach as the unauthorised disclosure released the information from the control Department of Families, Fairness and Housing (DFFH) to OpenAI overseas organisation. The investigation into the incident showed that nearly 900 employees across DFFH had accessed and used ChatGPT, representing almost 13% of its workforce.
In September 2026, a Sydney man was accused of using ChatGPT to help write scraper code that pulled roughly 9,000 sensitive documents faced court. The documents accessed included apprehended violence orders and details about minors from the NSW Online Registry.
Under the Australian Privacy Act1988 and the Australian Privacy Principles, entering someone else's personal information into a public AI tool is not a grey area, it is a disclosure. APP 6 requires personal information only be used or disclosed for the purpose it was collected for, unless the individual consent for a secondary purpose if an exception applies. In both cases no consent was provided by the individual for secondary purposes.
Additionally, OpenAI’s Privacy Policy states that training on conversation data is enabled by default in their bid to improve the model for everyone unless opted out of training (which most individuals do not pay attention to). OpenAI does not publish a guaranteed processing region for personal/free accounts, data residency/region selection is only available for the Enterprise-tier accounts.
Note, this is not an argument against using Generative AI, it's an argument for using AI securely, deliberately, and with proper guardrails. With majority of organisations lacking the governance structures and resource to manage AI or detect shadow AI, they leave themselves open to security, operational and reputational risks rarely involve sophisticated attacks. Perhaps if your organisation is unsure of the AI tools used by staff and what is flowing out into public, it is time to have a conversation before a regulator has it for you.
What can you do?
- Map the flow – Engage your IT,Security or Managed Security Service provider to understand the securitycontrols and configuration that provide visibility over what information isactually being entered into AI tools across the organisation.
- Draw the line – Establish governancearrangements that sets clear expectations for appropriate and acceptable AI use,ensuring that policies articulate not only what is permitted but the why.
- Train with context – Deliver targeted awarenesstraining that equip staff with the knowledge to understand the privacy andsecurity risk and impacts of their AI usage.
- Validate independently – Conduct anindependent assessment to evaluate the effectiveness of existing securitycontrols and identify residual gaps that increase exposure to AI privacy risks.
