You Cannot Defend What You Cannot Inspect
You Cannot Defend What You Cannot Inspect
Introducing CIRCUIT: an open-source framework for AI interpretability governance
By Eric Zielinski (Jumpmind)
July 14, 2026
We are deploying AI we can't explain, defending AI we can't inspect, and trusting AI<br>we can't audit. That is not a governance program. That is a liability surface.
The first AI governance committee meeting I sat in, somebody asked the product team a question<br>nobody had a real answer to. It wasn't a gotcha. It was the most basic question a security leader can ask:<br>when this model makes a bad decision, how will we know which part of the model made it?
The answer was silence, followed by a slide that said “human in the loop.” The loop was a reviewer<br>approving or rejecting outputs with no visibility into how they were produced. The model was a third-<br>party API. The vendor questionnaire had a row that said “model is proprietary.” We had a risk tier, a<br>registry, a vendor red team report, a SOC 2 Type II, and no idea what was inside the box.
That is the state of AI governance in most enterprises today. We have built extensive paperwork around<br>the outside of a black box and declared the box governed. The paperwork is real work. The box is still a<br>box.
At FIRSTCON26 I am releasing CIRCUIT as an open-source framework to close that gap and I am looking<br>for contributors from across the security community to build it out. This post explains the gap, what<br>CIRCUIT does about it, and why now.
What our existing controls actually do
If you have stood up an AI program in the last two years, you probably have a risk tiering scheme, a<br>model or agent registry, output monitoring with guardrails and DLP, a vendor questionnaire, a red team<br>program, and a mapping to NIST AI RMF. Good. Do not throw any of that away.
But look at what those controls actually do. Risk tiers classify intended use. Registries track where a<br>model is deployed. Output monitoring fires on patterns in what the model says. Vendor attestations are<br>assertions about the vendor's process, not the vendor's model. Red teaming probes behavior from the<br>outside. Every one of these controls treats the model as an opaque function from input to output.
When an AI-driven SOC triage tool buries a real intrusion as a false positive, those controls tell you the<br>model was "high risk, deployed in prod, monitored, vendor attested." None of them tell you which<br>feature the model weighted wrong, so your incident responders can't tell whether it was a one-off miss<br>or a blind spot an attacker can reproduce on demand. When a coding assistant slips a subtle backdoor<br>into a pull request, none of them tell you whether it came from a training artifact, a poisoned<br>dependency, or a prompt injection pathway in the repo the difference between a code review finding<br>and a supply-chain compromise. When an autonomous response agent quarantines the wrong host or<br>pushes a bad firewall rule mid-incident, none of them tell you why it made that call, leaving the blue<br>team to reverse-engineer their own tooling while the clock is running.
Your existing AI governance answers who owns the model, where it runs, and whether anyone<br>approved it. It does not answer why the model did what it did. Only one of those questions matters<br>during an incident.
Seven things a CISO is now accountable for and can't answer
Explainability under incident pressure. When the post-incident review asks why a detection<br>model missed an intrusion or flagged the wrong activity, regulators, auditors, and your own<br>board want the reasoning — not a log line proving the output was recorded.
AI-augmented adversaries. Attackers are using generative models for phishing, voice clones,<br>and synthetic identities at scale. Your defensive models are the counter — but defending with<br>them requires understanding what they key on, because the adversary will probe for that and<br>evade it.
Data and model poisoning. Training-time compromise of weights and fine-tune corpora is a<br>supply-chain attack on your detection stack. It is detectable only by inspecting internal<br>representations, not by watching outputs.
Failures in high-consequence response workflows. When an AI tool drives containment,<br>enrichment, or alert triage and confabulates, remediation requires knowing which part of the<br>reasoning went off the rails — not just that the SOC acted on bad information.
Inherited vendor AI in your defensive tooling. It is already in your SIEM, your EDR, your code<br>host, and your collaboration stack — Atlassian, Slack, GitHub, Salesforce, Microsoft 365 —<br>whether you adopted it deliberately or not. You own the blast radius. The vendor owns the<br>weights.
Prompt injection against security agents. MITRE ATLAS AML.T0051. An AI agent reading<br>attacker-controlled content — a malicious log entry, a poisoned ticket, a crafted email in an<br>inbox it monitors — can be turned against you. Defenses that don't understand the model's<br>internal response to...