Citation Isn't Causation: A Meter for What AI Answers Are Actually Made Of

I've spent about twenty years building software, and the last few of them living inside agentic AI — the systems that go fetch documents, write you an answer, and then hand you a tidy little list of "sources." Those lists have bugged me for a long time, and it took me a while to say why out loud. A citation is a claim. It's the model telling you, after the fact, what it used. Nothing checks it against what actually happened inside the answer. We just take its word for it.

I've gotten allergic to taking things at their word — in business, in who I choose to work with, and apparently in software too. So this is the thing I built to stop taking the citation at its word. It's called Tribute, and this isn't really a launch post. It's more honest as a log of getting the idea wrong a couple of times before it got useful — because the getting-it-wrong is the part I'm actually proud of.

Citation is not causation

Ask a RAG system a question and it hands you an answer plus a list of cited sources. That list is the model's own account of what it used, and nothing forces it to be true. In practice a source can sit right at the top of the citation list and contribute almost nothing to the actual text, while another source the model never names is quietly doing the real work.

Tribute tests the claim instead of trusting it. For a given answer, it removes a source, regenerates, and measures how much of the answer actually collapses. Do that across every source and you get a causal picture of what drove the answer — not a self-reported one. It also flags the case where you can pull every source out and the answer barely moves, which means the model was mostly answering from its own trained-in memory rather than anything you handed it. Every result gets written to a tamper-evident record, so the measurement itself can be checked later instead of just trusted in the moment.

I want to be careful about what that flag means, because it's easy to over-read. "Parametric" doesn't mean wrong, and "grounded" doesn't mean right. A model answering from memory can be perfectly accurate — plenty of facts are true without needing a citation. So this isn't a lie detector. It's a way of seeing whether an answer's real support structure matches the one it presented to you, so you actually know what you're vouching for. If you're a publisher, or an enterprise trying to understand what an AI answer inside your own stack is really resting on, that's the whole ballgame. Not compliance — defensibility. Knowing what you're standing behind before you have to stand behind it.

The one-dollar test that changed the plan

Before I built anything else on top of the grounding number, I ran a roughly one-dollar kill-test: fifteen trending questions through the engine, just checking whether the number actually moved from one question to the next. It didn't. It was a flat line.

A flat line is not a signal, and honestly that's the part of this story I find more interesting than what I'd been planning to do with it. The flatness wasn't a bug — it was structural. Our own retrieval setup pushes the model to answer from the provided sources by construction, so grounding was always going to read high almost everywhere I pointed it. Anything built on top of that number would have been built on a constant. The lesson isn't complicated — run the cheap test before you build the thing, not after — but it's one I seem to have to keep relearning. It's a small act of not lying to yourself, and those add up.

What the same test surfaced instead

The kill-test wasn't a dead end, though. Staring at those flat grounding numbers, I noticed the signal with teeth was sitting right next to the one that didn't have any. It wasn't how much of an answer was grounded — it was whether what the model cited lined up with what it had actually used. Cited-but-inert sources, and sources doing real work with no citation at all, kept showing up. Grounding was flat; the mismatch between citation and causation was where the interesting variation lived. Which raised the obvious question: do different models differ in how faithfully they cite? That's a hypothesis worth testing — not something the kill-test had already settled.

So I pointed the whole demonstration somewhere better — a benchmark. Give three model families the exact same sources for the exact same question — Claude, Gemini, and GPT — and measure which ones cite what they actually leaned on versus what they merely listed. To keep it honest and cheap, I drove the Codex and Gemini command-line tools headless as the model backends instead of paying for API access just to run the comparison.

I want to be precise about where this stands, because it would be easy to dress it up. I do not have a finished leaderboard, and I'm not going to tell you which model "wins." What I've got so far is smoke-test scale — single-question runs proving the pipeline works end to end across all three families. The method is real; the publishable numbers aren't in yet, and I'd rather say that plainly than round up. One more boundary worth stating: this measures models running over sources I control, in a pipeline I built. It is not an audit of ChatGPT.com or Perplexity's live black box. I can't see inside those, and I'm not going to pretend otherwise.

A metric bug, and where game theory came in

Partway through I found a real problem with the scoring — a technical one, not a philosophical one. Strict "remove one source and see what breaks" ablation under-credits redundant sources. If a fact shows up in three sources and you pull any one of them, the answer barely changes, because the other two still hold it up. So each of the three scores close to zero, even though together they're obviously what the answer is built on. The metric was doing exactly what I asked and still painting a misleading picture.

The fix came from cooperative game theory: Shapley values. Instead of asking what happens when you remove one source, Shapley attribution asks how much each source contributes on average across every possible combination it could appear in. Redundant sources end up fairly sharing the credit instead of each getting shortchanged for having backup. And a cleaner grounded-versus-parametric number fell out of that math for free, with nothing bolted on. I didn't go looking for game theory — the wrong number forced it — and the fact that it's a well-worn tool rather than something I cooked up myself is exactly why I trust it more than my first attempt.

What the meter actually shows

The demo is the fastest way to see the gap I've been describing, because you can watch it open. Every answer gets scored by four different backends and you can switch between them live: retrieval rank, the model's own citations, semantic similarity, and the causal one. The first three are passive — they observe what was available, or what was claimed. Only the last one intervenes.

Start on the naive backend and it reports 100% grounded, because everything retrieved is assumed used. Switch to causal and watch the number move. That toggle is the whole argument in one gesture: same question, same sources, same answer — and a confident number comes apart the moment something actually tests it.

There are four scenarios in there and each one breaks a different assumption. A clean baseline where all the backends roughly agree, so you can see the thing isn't just always crying foul. A distractor source, where self-reporting overpays. A case where the model answered from its own trained-in memory and no source deserves the credit. And a redundancy case — the one the Shapley work above exists for. For each source you get its contribution broken into the parts that produced it: relevance, authority, uniqueness, and actual usage. Plus a line for how much of the answer nothing you supplied can explain, which is usually the uncomfortable one. Every run writes to a hash-chained record you can verify, and that matters more than it sounds — a measurement nobody can re-check is just another claim, which is the exact thing this was built to stop taking at face value.

If you'd rather not take the canned examples at their word — which, given the subject, would be fair — there's an open prompt that retrieves live sources and runs the causal path against them — and, on the way, goes looking for whatever licensing terms those sources actually publish.

Who ends up needing this

Two groups, for opposite reasons.

Publishers are the cleaner case, because money is attached. As AI answers replace clicks, who gets paid for the content behind an answer stops being a philosophical question. Every compensation scheme I've seen routes payment through some attribution signal — and if that signal is the model's own citation list, publishers get paid for being cited rather than for being used. Those are not the same set. A source can sit at the top of the citations and contribute nothing; another can carry the answer and never be named. Pay on self-report and you overpay the first while stiffing the second. The research has converged on the same spot — several 2025–26 papers frame fair publisher compensation as precisely this attribution problem, and more than one of them reaches for the same cooperative-game math I ended up at.

This is where RSL matters, and it's the part I'd point a publisher at first. Really Simple Licensing is the emerging standard for declaring machine-readable terms: a publisher publishes an rsl.xml, discoverable from robots.txt, saying what AI systems may do with their content and how payment works — per use, or on an attribution basis. Something like fifteen hundred publishers have endorsed it. Actual deployment is barely underway; as I write this, Stack Overflow and the RSL Collective are about the only places shipping real terms, which is why the rates in my demo are labelled illustrative rather than dressed up as a market that exists.

But here's the thing that made me build toward it. RSL declares the payment and leaves the measurement blank — per-source attribution is simply undefined in the 1.0 spec. A publisher can say "pay me per use," and nothing in the standard answers how much did this source contribute to this answer. That undefined field is the number I've been measuring this whole time. So the demo emits a settlement record in RSL's shape — license reference, attribution score, usage, amount, the method that produced it, and a hash of the response it came from — and the honest caveat rides along with it: the rate is illustrative until the publisher's own license server resolves a real one. I'm not trying to replace the rail. The rail is the good news. I'd just rather the number underneath it wasn't computed by whoever is writing the cheque.

The broader move is from access to attribution. The first wave of this fight was about crawling — can you take it, and what do you pay for the right to. The next one is about use, and use is a measurement problem: who gets paid depends on what actually contributed. Access is a door you can lock. Contribution is a quantity somebody has to measure, and it should not be the party with the incentive.

It's worth being precise about what's being measured, because a competing definition is already in the field. Some approaches estimate a publisher's aggregate influence on the model from training — how much your corpus shaped the weights. That's a different quantity. This measures one specific answer, at inference time, against the sources actually put in front of it. Both are defensible things to want. They are not interchangeable, and it matters a great deal which one a cheque gets written against.

The enterprise case is less about revenue than exposure. If you're running RAG in medicine, law, or finance, sooner or later someone asks which sources are behind a particular answer — and "the model listed these" is a thin reply when the question is adversarial. Not compliance; no regulation currently forces this, and I'd rather not pretend otherwise. Defensibility. The difference between believing your system was grounded and being able to show it, months later, to someone with no inclination to take your word for it.

Holding the code to the same standard as the answers

Every change in this project went through review from two reviewers on different model families — one Claude-based, one GPT-based. That wasn't box-checking. Different model families have uncorrelated blind spots, and running the same diff past both caught real edge-case bugs a single reviewer waved through. The whole premise here is don't trust one model's account of itself — so it would have been a little absurd to then trust one model's review of the code that enforces exactly that. You have to hold the thing to its own standard, or it's just a slogan.

Where it stands

Tribute is live at tribute.staqs.io, and the source is on GitHub at github.com/ecgang/tribute.

The benchmark is still early, the numbers are still smoke-test scale, and I'd rather ship it honestly than dress it up. Next is running the cross-model comparison at real scale instead of one question at a time, and seeing whether the early signal holds up once there's enough data to actually mean something. I'll write that up too — flat line or not.

If any of this is something you actually deal with — you're paying for content that AI systems consume, or you're the one who'll be asked to justify an answer your system produced — I'd like to know whether the meter tells you anything you didn't already know. Go break it, and tell me where it's wrong.