A plain-language guide to Explainable AI — and why “ trust us” is no longer good enough
Mohan has worked at a garment factory in Surat for four years. Steady job, regular salary, nothing missed. When he and his wife decided to open a small stationery shop back home in Odisha, he did what millions of Indians do now — he applied for a digital loan through his phone.
Rejected.
He tried again. Different lender. Rejected again.
Nobody could tell him why. The app didn’t explain. The customer care line read from a script. The algorithm had looked at his data, made its decision, and moved on. Final, silent, unappealable.
What the algorithm likely couldn’t process: Mohan moves between two states. His SIM is registered in Odisha but he works in Surat. To a credit scoring model trained on urban, sedentary data patterns, that geographic mismatch reads as a red flag — not because Mohan is a risk, but because the model was never designed to understand someone like him.
This is not an edge case. India has over 300 digital lenders, and most of their models are built for urban, employed, digitally traceable borrowers. The hundreds of millions who don’t fit that profile get a rejection and no explanation.
Just a closed door.
We built something we can’t read
Here’s the uncomfortable truth about modern AI systems: the more powerful they get, the harder they are to explain.
The models driving today’s most consequential decisions — credit scoring, medical diagnosis, fraud detection, welfare eligibility — aren’t simple rule-based systems you can trace with a flowchart. They’re neural networks trained on vast amounts of data, finding patterns so layered and complex that even the teams who built them can’t always tell you why the model landed where it did.
This isn’t a bug. It’s an architectural reality.
The very thing that makes these models powerful — their ability to find non-linear patterns across millions of variables — is also what makes them opaque. You get the answer. You don’t always get the reasoning.
For low-stakes decisions, that’s manageable. If a recommendation engine suggests a film you end up not enjoying, no harm done.
But when the decision is whether you get a loan, whether you’re flagged as a fraud risk, whether a welfare system decides you’re eligible for food — “the model decided” stops being acceptable.
When the system can’t see you
The Aadhaar story is one most Indians know in outline. The details are harder to sit with.
Nearly four crore ration cards were cancelled across India because they could not be biometrically linked to Aadhaar — many without prior notice, and with no clear path to appeal. The system had made a determination: a failed fingerprint scan, a data mismatch, a linking deadline missed. Eligible families lost access to subsidised food. Not because they were ineligible. Because the system said so.
For elderly beneficiaries in rural areas, the situation carried a particular cruelty. Some walked miles to ration shops, only to be turned away when worn fingerprints failed to authenticate. They weren’t ghosts. They were just people whose hands had worked too hard for too long for the scanner to recognise them.
The algorithm wasn’t malicious. It was probabilistic.
A citizen’s identity had become a statistical score — above the threshold, you exist; below it, you don’t.
When families asked why they were denied, there was nothing the system could tell them. Not in terms they could act on. Not in terms that would say: here’s what went wrong, here’s who to call, here’s what would need to change.
The black box didn’t just make a consequential decision. It made an unexplainable one.
“Trust me” has never been a good enough answer
There’s a principle in medicine most of us take for granted: informed consent. Before a procedure, a patient has the right to understand what’s being done to them and why. Explanation isn’t a courtesy — it’s foundational to the relationship.
We’ve somehow built an entire generation of AI systems without applying the same standard.
Think about it from the receiving end. When a doctor recommends treatment, you expect to understand why. What the markers showed. What the evidence supports. You’d ask questions and expect real answers.
The model says so. I can’t tell you more than that.
That’s not medicine. That’s delegation — to a system nobody can interrogate.
This is the gap Explainable AI is trying to close. Not by making AI simpler — that ship has sailed — but by building tools and practices that let humans understand, interrogate, and where necessary, challenge what a system is telling them.
The methods quietly changing this
XAI isn’t one thing. It’s a growing toolkit — and one part deserves more attention than it gets.
Beyond identifying which data features influenced a decision, some approaches now generate contrastive explanations: not just why this outcome, but what would have needed to be different for the result to change. For Mohan, that’s the difference between a silent rejection and something like: your application was flagged due to a geographic mismatch in your registration data — here’s what that means and what you can do. One is a wall. The other is a door, even if it’s not yet open.
This shift is gaining regulatory weight. The EU AI Act mandates explainability for high-risk AI applications — a legal obligation, not a design preference — with enforcement timelines that land in 2026. Indian enterprises with European exposure are already in scope.
Closer to home, the RBI has been signalling the same direction, pushing banks, NBFCs, and fintechs to ensure their credit algorithms are regularly tested, auditable, and not treated as black boxes that answer to no one. The regulatory conversation in India is no longer theoretical.
The organisations paying close attention aren’t just the ethically minded ones. They’re the strategically minded ones. In regulated industries, the ability to explain an AI decision is quietly becoming the difference between winning and losing a contract.
The question underneath the question
Here’s what I keep coming back to when I think about XAI.
We don’t want explanations just because regulations require them, or because they help catch bias, or because they build trust — though all of that is true. We want explanations because explanation is how humans extend trust in the first place.
Think about every professional you’ve genuinely trusted — a doctor, a financial advisor, a mentor. What earned that trust wasn’t just that they gave you good answers. It was that they could show their reasoning. They could be questioned. They could be wrong in ways you could identify and correct.
An AI system that delivers verdicts without reasoning isn’t a trusted advisor. It’s an oracle.
And the problem with oracles isn’t that they’re always wrong — it’s that you have no way of knowing when they are.
Mohan deserved an explanation. Not a technical readout. Not a disclaimer buried in the app’s terms. A real answer, in plain language, that told him what had been weighed, what had tipped the balance, and what he could do about it.
The same goes for every family that walked to a ration shop and walked back empty-handed.
It’s actually the bare minimum.
COMING UP NEXT
Next issue, we put Responsible AI and Explainable AI side by side — because the question of who’s accountable and the question of who can explain are, it turns out, the same question wearing different clothes.
