Why AI can’t give you the same answer twice

And what we built to fix it.

I ran a test I want more people in finance to run for themselves.

I took a standard question, a 5-year levered IRR on a $7M commercial real estate deal, and asked the best AI model on the market to solve it. Then I asked it again. Same question, same inputs, ten times in a row.

I got ten different answers.

The correct figure is 10.8463781499708%. Claude (Opus 4.8, thinking mode, medium effort) landed on it zero times out of ten. Its answers ranged from 10.84% to 10.8552%. The spread is small, and if that were the whole story I might let it go. It is not the whole story. The model used a different method on almost every run. Ten questions, ten approaches, ten numbers that did not agree with each other.

This is not a prompting problem

We’ve heard the objections: “it’s a prompting problem.” “The LLM’s will catch up soon enough.” “Why does this matter?” “You can just roll your own with a python math library and Claude Code.” OK.

Large language models generate an answer by predicting a likely response one step at a time. When you hand one a calculation, it does not retrieve a fixed procedure and run it. It reconstructs an approach on the spot, from scratch, every time you ask. A better model reconstructs a better approach and narrows the range of wrong answers. It does not make the approach repeatable. You are still sampling from a distribution. Ask twice, and you can get two methods and two results.

That behavior is a feature when you are brainstorming or drafting. It is a liability the moment someone is going to sign a document based on the number.

And saying you’re going to build it yourself out of python libraries is certainly an approach. When’s the last time you (or anyone in development) rolled your own payment clearing code? Why not?

The trust tax

Here is what that liability costs in practice, and I suspect it will sound familiar.

Every finance team I know that has adopted AI still double-checks everything it produces. The tool is fast, so the analyst gets an answer in seconds, then spends the next twenty minutes verifying it by hand, because you cannot defend a number you cannot reproduce. The speed you were promised gets eaten by the checking.

And underneath the AI question is the older one that never went away: which version of the spreadsheet is the real one? The model your colleague built, the one with the tweaked assumptions, the copy on the shared drive from last quarter. Reproducibility was a problem before AI. AI added a faster way to produce numbers you cannot reproduce.

A different route

We built TrueMath on a different starting assumption: that you want your calculations to use your business logic, the same way, every single time.

You define a domain, the set of calculations and the logic your work depends on, or you take one from our Public Library and adapt it. Then you ask your questions in plain language, the way you would ask a colleague. Behind the conversation, a calculation based on an engine with 30 years of development maturity runs the math. One method, defined once, applied identically on every run. Same question, same answer, every time. And every result comes with a full trace, so it is auditable, sharable, and defensible.

That is what “with receipts” means. It shows the work automatically, every time, in a form you can hand to a colleague, a client, or an auditor.

Yes, I concede: LLMs will get better

Right now, TrueMath is faster, cheaper, more precise, and far more energy efficient than pushing the same question through an LLM. In our test harness, running the same prompt ten times each, TrueMath through a natural-language layer came back about 9.6x faster and roughly 500x more energy efficient per calculation. Asked in structured form, it is faster still. (Through a direct-to-TrueMath-structured-prompt. method: TrueMath is 70x faster, 750x more energy efficient.)

We don’t expect those gaps to last forever. They’ll last for a good long time, but LLMs will improve. So I want to be clear about which advantage is temporary and which is not.

Speed, cost, and energy are today’s edges. They are real, they matter, and they may erode.

Determinism is a different kettle of fish. TrueMath uses your business logic, only your business logic, the same way on every run, because that is how it is built. No amount of model progress closes that gap, because it is architectural, not a matter of tuning. A better LLM is still an LLM reconstructing the method each time. That is the part I believe holds forever.

Try it

TrueMath is live. If your work depends on a number being both right and reproducible, I would like you to put it through its paces and tell me where it holds up and where it does not.

You can set up an account at app.truemath.ai/signup. It is free, and I read the feedback personally.

Business at the speed of conversation. With receipts.

Bill Kelly Co-founder and CEO, TrueMath