Debugging is harder than writing new code. When debugging, you’re trying to figure out what others have done and how it fits together. When writing new code, you don’t face those same challenges.
Debugging AI is even more difficult because, unlike traditional debugging through deterministic means, AI is probabilistic. This means getting two results from two runs isn’t necessarily wrong. Further, instead of AI throwing some hard error, it packages up a response and displays that as if it’s truth.
If you’re old school, you’re familiar with setting breakpoints, watch variables and stepping through code line by line. If you’re even older school, there was the well placed print statements to identify value or state. While debugging can be hard, in deterministic programming we have a toolbox filled with useful ways to trace through code. Things are different with AI.
Normal Stages in Debugging
AI aside, the following may look very familiar. For the non-technical folks, this will give you some insight into how a programmer works through issues.
As you can see above, a programmer is assigned a bug to fix. Here are the stages.
- That can’t happen!
- That doesn’t happen on my machine
- That Shouldn’t happen
- Why does that happen?
- OH, I SEE
- How did that ever work?
Now here’s the thing. Debugging tools haven’t caught up to AI (we’ll see below what we do have) We don’t have the ability to just simply duplicate, trace and fix.
Two Basic Approaches
During my working years, I generally took one of two approaches. Sometimes even combining them both. They are:
- Inside out – this approach starts at the lowest level (the problem) and works backwards to identify the error.
- Outside in – this approach starts with the input and work towards the problem.
These same approaches can be used with AI, they just cost more in terms of hardware and software requirements. They are generally considered heavyweight tools requiring a substantial investment.
Inside Out
In AI identifying the trouble spot is not really a line of code, but rather a neuron or set of neurons (clusters). In a previous article I wrote about Mechanistic Interpretability and Sparse Autoencoders. The science, mechanistic interpretability officially came about in 2020, with sparse autoencoders becoming the breakthrough tool in 2023. As you can tell, this is a young and growing field. The processing requirements are still quite large, but like most technology, I see this getting greatly reduced in the future. What I want to mention here is that the tool can now achieve the outside in approach to debugging. Now we can identify the neurons or group of neurons that are causing faulty output.
Outside in
To address debugging from the outside in means to start as far out as we can get (the actual training data). To use this approach there are something called Influence Functions. This requires a fair amount of math that I won’t bog you down in. For those interested in the nitty gritty, check out LessWrong in their article Influence Functions, Why, What, How. What the influence functions allow us to do, is take a hard look at each data set being used in training.
Combining The Two Approaches
When an AI model produces a bad output (hallucination, safety policy violation or biased decision), we can combine the two approaches above into a 3 step loop.
- Isolate the internal mechanism (Inside out using Sparse Autoencoders or SAE’s).
- Instead of staring at thousands of raw, polysemantic numbers you run an SAE to translate those numbers into concepts.
- Example: You notice the model outputted toxic advice. The SAE shows that a specific Feature (“Sarcastic dismissiveness”) and another Feature (“Medical advice”) fired simultaneously.
- Trace the Origin (Influence Functions / Outside-In)
- Now that you know the exact internal feature causing the bug , you use Influence Functions targeting that features activation (or the final bad token) to query the pre-training or fine tuning dataset.
- Example: The influence function points to 15 specific scraped forum posts in the training set where medical advice was delivered with heavy sarcasm.
- Intervene and Patch (Closing the Loop)
- Once you have the complete flow, you have options for how to fix the problem:
- Internal Patching: Clamp or steer the Feature down via activation editing (fast, inference time fix).
- Data Patching: Remove or re-weight those 15 specific forum data points and re-fine-tune the model (permanent root-cause fix).
Summary Checklist
| Dimension | Inside-out (SAE’s) | Outside-in (Influence Functions) |
| Focus | Internal activation space (latent representation) | External dataset space (training dynamics) |
| Target | Features, monosemantic neurons, circuits | Data samples, documents, fine-tuning examples |
| Metaphor | An MRI Scan: Shows which brain circuits are firing during a response. | A Background Check: Shows what past experiences caused that behavior. |
| Fix Type | Model steering, circuit ablation, activation clamping | Data filtering, dataset pruning, targeted unlearning |
What “IDE Integration” Looks Like Today
While not perfect due to the overhead of using SAE’s and Influence Functions, there are 3 main buckets used today.
- Vector Search as “Lightweight Influence”:
Instead of running mathematically pure Influence Functions, developers use vector databases (like Pinecone, Qdrant, or Chroma) to index training/RAG datasets. When a model hallucinates, the “debugger” performs a fast cosine-similarity search on embedding space. It isn’t exact causal influence, but 80% of the time, it points you directly at the bad source document. - Feature Dashboards (SAEs as “Profiler Views”):
Tools like Anthropic’s open-source circuits visualizations or Neuronpedia act like a memory profiler. You don’t inspect raw weight tensors; you inspect a visual graph of high-level feature activations firing in real time for a given prompt. - Observability & Tracing (LangSmith, Arize, Phoenix):
Right now, most AI “debugging” in production is standard execution tracing—logging prompt chains, retrieval steps, agent loops, and token-level log probabilities. It’s essentially stack tracing for LLM orchestrations rather than the model’s brain.
What it All Means
For the casual observer, we don’t really care about the math or the debugging attempts. What we care about are the finished products. Are they stable, reliable, supportable? Do they solve the problem we’ve intended for them?
In my last article titled Nail The Present to Earn the Right To Ship the Future, I wrote about what AI is currently doing well and what must get better. A big part of getting better is to more fully understand and having the ability to debug the model.
Summary
We can and will gain ground within the AI space. We just need to relax, let things unfold instead of racing to market with systems we don’t understand and cannot debug. The technology is to important to let only money drive the decision making process.
To fully explain things like debugging AI, mechanistic interpretability, sparse autoencoders and influence functions would take hundreds of pages for the math alone. If you’re a watcher of AI, or perhaps a manager or executive working in an organization evaluating AI, you don’t need to know the math. What you should be more focused on are the features, reliability and the means to support the model.
It’s funny and scary to think not many years ago we accepted answers like ‘I don’t know’, or ‘that’s what the system did’. Those days are over (finally). As we learn more and better ways to debug the AI models, you’ll start to see real improvement and rollouts. For now, stick to what works. Read my article Nail the Present to Earn the Right to Ship the Future .
Go ahead, get your hands dirty, start using AI, but do so with a level head.
What do you think? Do you think it’s time for lightweight debugging tools? I think it’s about time we look behind the curtain. Our systems will be better for it.
Drop me a line and let me know your thoughts.