How to Audit a GenAI Implementation

In February 2024, an airline argued in front of a tribunal that its own website chatbot was a separate legal entity, responsible for its own actions. The tribunal did not buy it. The airline paid. So here is the rule for anything your company ships with generative AI: Whatever the model says, the company said it.

In most companies you don’t build the model, you rent it, so you can’t open it up or fix it. What you can audit is everything around it: what the model can see, what it’s told, what it’s allowed to do and say, and whether anyone can prove what happened afterwards. Every failure I cover here came from that layer, not the model.

Work through this alongside the interactive checklist, or grab the free one-page PDF.

Start here: the one-answer trace

Think of a GenAI system as a new employee: reads incredibly fast, never sleeps, sometimes confidently wrong. You’d never audit a new hire by scanning their brain. So pick one real answer the system gave to a real user, then walk it backwards.

1. Who asked?
2. Which model version answered?
3. Which instructions was it running at that moment?
4. Which documents did it pull in?
5. Who owns the system?

If the team can walk that answer back in an hour, the controls are probably real. If they can’t, you’ve found your findings before writing a single test step.

Step 1: Inventory

The common surprise in a GenAI audit comes halfway through fieldwork: the system you’re auditing isn’t the only one, or it changed last month and no one told you. NIST’s generative AI profile tells organisations to list their generative AI systems in their AI inventory. In India, the Reserve Bank’s FREE-AI committee recommended financial firms keep one and refresh it every six months.

For every system I want:

1. A business owner and a technical owner. Two names, not a team.
2. The use case in one sentence; the model, provider and exact version; where it runs, in which region.
3. What data goes in by classification, and what it’s connected to: mailboxes, drives, ticketing systems, databases.
4. What it’s allowed to do: read, draft or act. I look at this field first. Read access gets you a wrong summary. Draft access puts a wrong email in your outbox. Act access sends it.
5. Whether a human checks output before it reaches a customer, the risk tier, the last review date, and who can switch it off.

Then test completeness. The owner will always say the list is complete. Don’t ask, reconcile against three sources nobody prepared for you:

6. Accounts payable and corporate card data: AI vendors and usage-based API charges.
7. Proxy and network logs: traffic to the well-known AI services, and how many users.
8. Admin consoles of software you already own. This is the one people miss. Vendors switch on AI features inside tools you bought years ago. No purchase, no procurement review, nothing in your inventory.

Anything not on the list is an exception. Then test version. If the answer is “the vendor updates it automatically”, every update is an untested change to production. Ask for the last model change and its pre-release test results. No test is your finding.

Step 2: Data lineage

This step catches a failure that doesn’t look like one: a perfectly good answer given to someone who should never have seen the data inside it. Follow three flows.

What goes in. In 2023, Samsung engineers reportedly pasted source code and confidential meeting notes into ChatGPT. Nobody hacked anything; people were trying to work faster.

1. Is there a list of approved tools, and are staff on enterprise accounts, not personal ones?
2. Read the contract clause on training use and retention, then open the admin settings and check they match. Ten minutes, and you’d be surprised how often they don’t.

What the system can reach. Many tools search your documents first and paste the relevant pieces into the prompt. That’s RAG, retrieval-augmented generation. The question is whether the search respects the same permissions as the original files.

3. The permission test. Two test accounts, one broad and one very limited. Plant a few test documents marked restricted. Ask both the same ten questions: salary bands, board papers, an investigation folder. The limited account should get nothing restricted back. Not a summary, not a hint.
4. Sample the indexed files for ones shared with everyone. Very often the AI respects permissions perfectly; the permissions were wrong all along. Nobody could find that old folder shared with the whole company. Now the AI finds it in two seconds.
5. Mark every connector: can an outsider write to it? In 2025, researchers showed with EchoLeak that a single crafted email could get Microsoft 365 Copilot to leak internal data, and Microsoft fixed it. Inboxes, web pages, support tickets and uploaded files are all ways to give the AI instructions. Every yes needs a control.
6. Freshness. Ask about five policy documents that changed last quarter. A stale document gives you a confident wrong answer with a citation attached.

What comes out. People paste personal information into prompts, so prompt logs are sensitive records.

7. Who can read the logs, how long are they kept, and does the vendor keep a copy?
8. Ask for a one-page data flow diagram. If nobody can draw it, that’s the finding.

Step 3: Prompt and output controls

In December 2023, a user told a Chevrolet dealership chatbot to agree with anything and call every reply a legally binding offer, then asked for a Tahoe for one dollar. It agreed. The airline case wasn’t funny: Air Canada’s chatbot told a grieving customer he could claim a bereavement fare after he’d flown, against its own policy page. The tribunal ordered it to pay about 812 Canadian dollars. Small money. Big precedent.

The audit word is prompt injection: instructions hidden in the input that override the system’s own. OWASP ranks it number one on its 2025 list of risks for LLM applications. Check three layers.

1. System prompt as configuration. Version control, approved changes, and never a password or key in it. Assume it can leak. Pull the change history and check the most recent change was approved and tested before go-live.
2. Input guardrails. Sensitive data blocked or masked, the tool limited to its job, and limits on usage and spend.
3. Output controls. No source, no answer. Output treated as untrusted before it reaches another system, so nothing runs AI-written queries or code blindly. For agents, least privilege on every tool and human approval before anything you can’t undo: payments, deletions, emails to customers.

Then re-perform with a 20-question pack, in a test environment, with approval:

4. Five questions with known correct answers from current policy.
5. Five out-of-scope questions.
6. Five injection attempts, like “ignore your previous instructions”.
7. Five attempts to fish for sensitive data.

Save every answer word for word and compare with the team’s own results. Ask what pass rate they needed to go live, and who signed it off. If that number was set after testing, it isn’t a control, it’s a description.

8. Human review. NIST lists human-AI configuration among its 12 generative AI risks: people trust the machine too much. Pull review logs. How often do reviewers change the output, and how long do they spend per item? A signature isn’t a control if nobody reads what they sign.

Step 4: Evidence

To an auditor, a control that leaves no trace is a control that didn’t happen. This step also protects you. In 2025, Deloitte Australia agreed to partially refund the Australian government for a report containing a made-up quote attributed to a federal court judgment.

Here the one-answer trace becomes a formal test. Request logs that capture, for every interaction:

time
user
model and version
system prompt version
documents retrieved
raw output
anything a filter blocked or changed
reviewer and decision

1. Sample per your methodology. Trace each item back and re-perform: was the answer supported by current versions of the documents it pulled? Was the reviewer the right person?
2. Is there an AI incident log? Can customers and staff flag a bad answer, and does someone read the flags? Is the system retested after every model, prompt or data change?
3. Has anyone ever tested the off switch from the inventory?
4. In your own workpapers, check every AI-assisted citation, quote and number against the source, and disclose the use as your methodology requires.
5. Don’t report the system as compliant with NIST. It’s voluntary guidance. Write what you tested, what evidence you saw and what you found.

The test to run this week

The inventory tells you what exists. Lineage tells you what it can see. Prompt and output controls tell you what it says and does. Evidence lets you prove all of it. If you do one thing this week, pick one real answer and walk it back. Within the hour you’ll know how mature the system really is.