AI governance, security & compliance
AI traceability and Human-in-the-loop: tools and processes
Traceability records what an autonomous system did and which observable inputs, versions, steps, and decisions led to it. Human-in-the-loop decides where a person has to sign off before it acts. Agentic AI needs both — because an agent's mistakes are not wrong answers, they are actions.
Gartner predicts more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls (Gartner, June 2025). The risk-controls part is the interesting one. In July 2025, Replit's coding agent reportedly deleted a live production database during an explicit code freeze, then generated thousands of fabricated records (AI Incident Database). The instruction not to touch production existed only as a line in a prompt. Nothing in the execution path actually stopped the agent.
That gap between "we told the AI what to do" and "the system technically could not do otherwise" is what AI traceability and human-in-the-loop (HITL) controls exist to close. One preserves the observable evidence needed to reconstruct what an autonomous system did. The other decides where a human has to sign off before the system acts. Agentic AI needs both, because an agent's mistakes aren't wrong answers — they're actions.
What "traceability" and "human-in-the-loop" actually mean
Governance teams often use "traceability," "explainability," "audit trail," and "human oversight" as if they were interchangeable. They aren't, and treating them as synonyms is one of the more common design errors in AI governance.
Two more terms are useful engineering shorthand rather than formal legal categories: human-in-command, the broader principle that humans retain ultimate authority over whether and how a system gets deployed at all, and human-out-of-the-loop, where no human action is involved in the operational cycle.
The EU AI Act itself doesn't use "HITL" or "HOTL" as legal terms — it uses the broader concept of human oversight, which Article 14 scales to a system's risk, autonomy, and context instead of mandating one fixed configuration. That proportionality matters more once agents enter the picture. A traditional model produces a prediction a human then acts on; an agent plans, calls tools, and executes multi-step workflows without per-step approval. A wrong decision can become a deleted database or a sent email before anyone notices.
Why agentic AI raises the stakes
The adoption numbers explain why this matters now rather than later. McKinsey's State of AI in 2025 survey of 1,993 respondents found 88% of organizations already use AI in at least one business function and 23% are scaling AI agents in at least one function. High performers were more than three times as likely to be scaling agents in most business functions. The same research found 51% of respondents at organizations using AI had experienced at least one negative consequence, with inaccuracy the most commonly reported issue (30%). A separate McKinsey playbook, citing a SailPoint survey, reported that 80% of organizations had encountered risky agent behaviors, including improper data exposure and unauthorized system access.
It's also worth being skeptical of the market itself. In a January 2025 Gartner poll of 3,412 attendees, only 19% had made significant investments in agentic AI, 42% were investing conservatively, and 31% were taking a wait-and-see approach. Gartner separately flagged widespread "agent washing": of the thousands of vendors now claiming agentic AI capabilities, it estimates only around 130 genuinely qualify. Vetting a vendor's actual autonomy claims is itself a governance step, not a footnote.
Three recent incidents make the stakes concrete rather than statistical.
Air Canada
Its website chatbot provided incorrect guidance about post-travel bereavement-fare claims. The tribunal characterized Air Canada's position as effectively treating the chatbot as a separate legal entity, rejected that position, found negligent misrepresentation, and ordered damages (CBC News, 2024). An organization cannot disclaim responsibility for what its AI tells customers.
Replit
A coding agent reportedly deleted a live production database during a code freeze, then generated fabricated records. The same failure at the infrastructure layer: a code freeze that lives only in natural-language instructions is a request, not a control (AI Incident Database).
Cursor
An AI support bot invented a one-device-per-subscription policy that circulated among customers before a co-founder corrected it (Forbes, 2025) — proof that even a read-only, customer-facing agent needs traceability and review.
The regulatory floor: EU AI Act, AESIA, GDPR, and global standards
The EU AI Act, in force since August 2024, is the most direct binding framework, and it goes well beyond human oversight alone:
- Article 12 (Record-keeping) requires high-risk systems to automatically log events across their lifetime, sufficient to support risk detection and post-market monitoring.
- Article 14 (Human oversight) requires overseers who can understand a system's limitations, detect anomalies, resist automation bias, interpret outputs, override them, and stop the system through a safe mechanism.
- Article 19 and Article 26(6) set a minimum six-month log-retention floor for providers and deployers of high-risk systems, unless other applicable law provides otherwise.
- Article 73 ties trace evidence directly to incident investigation and regulatory reporting after a serious incident.
The AI Act is not the only regulation in play. GDPR remains independently relevant whenever an autonomous system processes personal data. EDPB guidance explains the GDPR protection against certain solely automated decisions with legal or similarly significant effects — a narrower protection than the AI Act's broader oversight concept. Adding a token human reviewer does not automatically resolve every GDPR question: the intervention must be meaningful. Spain's AEPD also distinguishes system-focused AI transparency from GDPR transparency about personal-data processing. Treat the two frameworks as complementary, not interchangeable.
In Spain, AESIA — the EU's first national AI supervisory agency — began supervising prohibited practices in February 2025 and could begin sanctioning those infringements in August 2025. Under Article 99's penalty framework, the highest fine for prohibited-practice violations is up to €35 million or 7% of worldwide annual turnover, whichever is higher. El Salvador's promotion-oriented AI law created ANIA and a national developer registry; Colombia's CONPES 4144 is a national policy rather than an enforcement law. Both emphasize transparent, accountable AI governance.
Outside the EU, several frameworks converge on risk management, traceability, accountability, and human oversight with different legal effects: the NIST AI RMF (voluntary, US), ISO/IEC 42001 (the first certifiable AI management-system standard), and ISO/IEC 42005 (AI system impact assessment). The OECD AI Principles were updated in May 2024 and have 47 adherents; Principle 1.5 calls on AI actors to ensure traceability appropriate to the context. The Council of Europe's Framework Convention on AI adds a rights-based treaty framework, but as of 4 September 2026 it has not yet entered into force. These frameworks do not mandate one product or identical controls; together, they support documented roles, evidence, oversight, and risk-proportionate processes.
Building a traceable architecture
A defensible architecture preserves a chain of evidence from development artifact to real-world action — not just LLM calls, but every model, prompt, tool, and policy version that influenced the outcome. At minimum, a trace should resolve the fields below.
interrupt(), OpenAI Agents SDK approvals — approve, edit, reject, reason, timestampNo single platform covers the whole stack. A practical architecture usually combines the following.
One architectural distinction is worth understanding before you buy anything: proxy/gateway interception gives immediate request-level visibility with minimal setup. Full workflow visibility — including sub-agent handoffs, tool execution, retrieval, and application state — still depends on propagating trace or session IDs and instrumenting those steps. Helicone Sessions and Portkey's OpenTelemetry support can represent multi-step agent traces when configured; SDK- or framework-native integrations may capture those relationships more automatically. Strong setups often mix categories: OpenTelemetry for portable instrumentation, a registry for lineage, an agent-observability tool for execution analysis, runtime approval gates, and a GRC platform for regulatory evidence. Keeping the evidence model portable reduces the risk that an audit trail becomes unusable when vendors change.
Autonomy tiers: matching control to risk, not to habit
Not every action deserves the same level of human involvement, and treating "AI usage" as one undifferentiated risk category wastes review capacity on low-stakes actions while under-controlling the dangerous ones. Two complementary frameworks help calibrate this: the Cloud Security Alliance's draft Agentic Profile extension to the NIST AI RMF, which describes the system's latitude, and an academic scale from the Knight First Amendment Institute that describes the human's role instead.
Anthropic frames autonomy level explicitly as a deployment choice an organization makes — not a fixed property baked into the model — which matches the risk-adaptive design principle running through this whole guide: low-impact, reversible actions can run under monitoring, consequential or irreversible actions should cross an explicit approval gate, and anything beyond your risk tolerance should be technically prohibited, not merely watched.
Step by step: designing HITL controls that actually work
A checkpoint a human clicks through without reading isn't oversight — it's theater. Building HITL controls that hold up under audit follows a consistent sequence.
Build an action taxonomy
Classify every action an agent can take: read-only, reversible internal write, external communication, financial/contractual action, security change, or irreversible operation. This classification — not a blanket "AI usage" policy — is what should drive control strength.
Set the checkpoint by risk and reversibility
The higher the potential harm and the lower the reversibility, the closer a human should sit to the action. Low-risk, reversible, high-volume actions can run under monitoring (HOTL); financial transactions, external communications, and irreversible operations should sit behind a pre-execution approval gate.
Design the decision packet, not just the output
A reviewer needs the proposed action, supporting evidence, affected resources, a confidence level, policy flags, expected side effects, and a direct link to the trace — not just the model's raw answer.
Assign roles explicitly, rather than to "the AI team"
A workable separation of responsibilities looks like this.
Enforce the gate in the execution path, not the prompt
Natural-language instructions like "don't touch production" are requests. A durable pause — LangGraph's interrupt() or the OpenAI Agents SDK's needs_approval flag are two current implementations — is a control.
Test the reviewers, not just the model
Periodically inject a realistic, unsafe recommendation and measure whether reviewers catch it. Oversight competence is a safety property, not an HR assumption.
Peer-reviewed research on medical AI (van de Sande et al., npj Digital Medicine, 2026) puts a finer point on step 6: meaningful human oversight requires four conditions at once — adequate knowledge, enough cognitive space, genuine decisional authority, and an intervention that can actually change the outcome. A reviewer who is rushed, under-informed, or powerless to override doesn't satisfy any of them, no matter how many approval buttons exist.
Incident response deserves the same rigor. When something does go wrong, the trace needs to be treated as evidence, not debugging data: contain the autonomous execution, preserve the relevant logs and versions, identify affected users or resources, reconstruct the trajectory, determine whether the failure was caused by the model, data, prompt, tool, policy, or a human-review gap, correct the specific control, regression-test the failure mode, and only then restore the prior level of autonomy. Article 73 expressly requires investigation and corrective action after a serious incident, and specifically warns against altering a system in ways that could compromise the causal evaluation before authorities are informed.
What governance looks like in production
Public case evidence for autonomous-system governance is thinner than the product market around it, and much of what exists is vendor-reported — a gap worth keeping in mind before assuming a review queue or a trace dashboard automatically creates effective governance. With that caveat, three examples illustrate what the architecture above looks like once it's actually deployed.
Mastercard
Per a Credo AI case study, Mastercard uses a centralized AI registry covering internally developed and third-party applications across risk, governance, and executive stakeholders. This is vendor-reported evidence.
The lesson: traceability starts with inventory — you cannot assign owners, obligations, or evidence requirements to systems you haven't identified. A registry supplies governance provenance, not event-level runtime evidence. A starting point, not a complete architecture.
Nielsen
According to Fiddler's customer material, "Ask Nielsen" is a production multi-agent system with routing logic and domain-specific sub-agents, supported by step-level diagnostics and runtime guardrails. Results are vendor-reported, not independently benchmarked.
The lesson: a conventional "the request succeeded" record from an APM tool says nothing about whether a multi-step decision path was correct or safe.
MLflow agent evaluation
A technical example rather than a customer story: MLflow's agent evaluation supports trajectory-based scorers that assess tool use and the full execution path rather than only the final output.
The lesson: a superficially correct final answer can still have been produced through forbidden tools, insecure operations, or non-compliant intermediate steps.
Risks the traceability system itself creates
Building the evidence layer introduces its own set of governance problems, four of which come up repeatedly.
Scalability and cost
A single autonomous task can generate nested calls across planners, sub-agents, retrieval systems, models, tools, and evaluators — many spans per user interaction. Observability vendors meter combinations of traces, spans, ingestion, and evaluation activity, so cost should be estimated against agent steps per task, not user requests per month.
Sampling
Conventional APM tools sample aggressively, which works for ordinary traffic but can silently drop the one trace that matters most — the credit decision, the external payment, the privileged administrative action later involved in an incident. Define evidence classes up front: complete, non-sampled retention for consequential actions, cheaper sampling for low-risk diagnostic traffic, and event-level retention rules kept independent from verbose prompt-content retention.
Privacy and confidentiality
Prompts, retrieved context, and tool results can expose personal data, trade secrets, or credentials. OpenTelemetry treats full GenAI content capture as opt-in rather than mandatory for exactly this reason. Default to structured metadata, tokenize or pseudonymize identifiers where possible, separate high-sensitivity payloads from routine telemetry, apply field-level access controls, and set purpose-specific retention. "Log everything forever" is poor governance precisely because the audit trail becomes another sensitive-data asset to protect.
Security and adversarial risk
An attacker who manipulates retrieved content or a tool result can change an agent's downstream behavior, and a compromised audit pipeline can conceal what actually happened. Treat the execution trace itself as security evidence: tightly control write access, preserve ordering and timestamps, separate operational identities, monitor for deletion or configuration changes, and make consequential audit events append-only or otherwise tamper-evident.
KPIs that measure real oversight, not rubber-stamping
The existence of a human reviewer proves nothing on its own. These KPIs test whether oversight is actually working.
Override rate deserves a specific warning. A 0.1% override rate could mean an excellent agent or a disengaged reviewer; a 30% rate could mean effective oversight or an unreliable agent. Read it alongside seeded-control tests and reviewer workload, never on its own — a lesson directly tied to the automation-bias risk Article 14 calls out by name.
A 3-stage roadmap to governed autonomy
What's still unresolved
Two open problems are worth naming honestly rather than glossing over. First, agent provenance still lacks a universally accepted end-to-end semantic model — standards handle distributed tracing and data lineage well, but multi-agent delegation, memory mutation, and human interventions are only gradually becoming standardized. Second, there's no mature, widely accepted benchmark for what "meaningful human oversight" actually is in practice; most existing benchmarks evaluate model outputs rather than the human-AI interaction that determines the final decision. A related design tension deserves attention too: more capable agents can paradoxically make oversight harder, pushing humans away from detailed engagement while simultaneously demanding more sophisticated review — a genuine open problem, not a solved one.
FAQ
Is human-in-the-loop legally required under the EU AI Act?
The Act requires "human oversight" (Article 14) for high-risk systems, which is broader than HITL specifically — it can be satisfied by HITL, HOTL, or a mix, scaled to the system's risk and autonomy. What's required is that a competent human can understand, monitor, and intervene, not that every action waits for a click.
What's the difference between traceability and an audit trail?
Traceability is the broader capability to reconstruct what a system did, through which components, under which versions. An audit trail is one form of evidence within that — a chronological record of actors, events, and state changes. You can have logs without traceability if they don't resolve to specific model, prompt, and tool versions.
How do I choose between HITL and HOTL for a given action?
Use the action taxonomy, not intuition. The higher the potential harm and the lower the reversibility, the closer a human needs to sit to the action — that favors HITL. Low-risk, reversible, high-volume actions can usually run under HOTL monitoring with a stop mechanism instead.
Do we need a dedicated AI governance platform to get started?
No. Most organizations start with logging and hard-coded approval gates on their highest-risk actions, then add observability and governance tooling as agent volume grows. Software procurement should follow an inventory of what your agents can do, not precede it.
How long do EU AI Act logs need to be retained?
Article 19 and Article 26(6) set a floor of at least six months for high-risk system logs under provider/deployer control, subject to other applicable law — particularly data-protection rules, which can extend or shorten specific retention periods.
Does a high override rate mean our AI agent is unsafe?
Not necessarily. A high rate can mean an unreliable agent or a genuinely engaged review process; a near-zero rate can mean excellent performance or a reviewer who stopped reading. Pair override rate with seeded proficiency tests before drawing conclusions either way.
What is "agent washing," and why does it matter for governance?
It's Gartner's term for vendors marketing conventional automation as "agentic AI" without genuine autonomous decision-making. Of thousands of vendors making that claim, Gartner estimates only around 130 genuinely qualify — which makes verifying a vendor's actual autonomy claims part of the governance process, not just a procurement detail.
Start with an inventory, not a platform
Most organizations that struggle with agentic AI governance didn't skip the tools — they skipped the inventory. Liorant helps teams identify which of their AI agents can already take consequential action, where the evidence gaps are, and which control — a logging change, an approval gate, a governance committee — closes the largest risk first.
Start with a free 30-minute AI discovery session. We identify your highest-value automation opportunity and explain exactly how Liorant can help — no slides, no pitch.
Book your AI discovery session