Explainer · · 6 min read
How we score every AI tool
The number on every tool page isn't a vibe. Here's exactly what the Solthar Score measures. And what it deliberately ignores.
By Manish
Every tool in Solthar carries a Solthar Score from 0 to 100. It is the first thing most people look at and the thing we get asked about most, usually by the people whose tool scored lower than they hoped. So here is the whole method in plain terms, with no black box and no hedging.
The short version: the score answers one question. If you had this job to do today, and you could only pick one tool, how close is this to the right answer? Everything in the method exists to serve that question, and anything that does not serve it stays out.
The four signals
The score blends four inputs, deliberately weighted toward the one that actually decides whether you keep using something.
- Output quality, weighted heaviest. How good the results genuinely are, judged against the best tool doing the same job rather than against some absolute ideal.
- Reliability. Uptime, consistency, and whether the same input produces comparable output twice in a row.
- Momentum. Is the team actively shipping, or has the changelog gone quiet for eight months?
- Value. What you get relative to what you pay, with the free tier counted honestly rather than dismissed.
Quality carries the most weight because it is the only one users cannot work around. You can tolerate occasional downtime and you can budget for a price rise. You cannot fix output that is not good enough.
Why we compare inside a category, not across
A transcription tool and an image generator do not share a scale. Asking whether Whisper is better than Midjourney is not a question with an answer, so we do not pretend to answer it.
Every tool is judged against the others doing its specific job. A 92 in voice cloning means something precise: this is close to the best available option for cloning a voice. It does not mean it outranks an 88 in video editing, because those two numbers were never measured on the same axis.
This is also why category leaders cluster in the high eighties and nineties. In a mature category several tools are genuinely excellent, and compressing them apart artificially would be inventing distinctions that do not exist.
What the score deliberately ignores
A directory is only useful if its rankings cannot be bought. That is not a marketing line, it is the entire product. The moment a score can be influenced by the company being scored, every other score on the site becomes worthless too, including the honest ones.
- Marketing budget. How loudly a tool is advertised has no bearing on whether it works.
- Funding raised. A large round signals investor confidence, not product quality, and plenty of well-funded tools have shipped nothing.
- Social following and launch-day hype. Both measure distribution, not usefulness.
- Whether we have any commercial relationship with the company. We do not sell placement, and if that ever changes it will be disclosed on every affected page before the change takes effect.
A tool with 400 users can outscore one with four million. Quality does not care about headcount.
Where the judgement actually lives
We should be straight about this: the score is not a measurement in the way a benchmark is a measurement. Reliability and momentum can be checked against public evidence. Output quality involves human judgement, and human judgement can be wrong.
What we can promise is that the judgement is consistent and that the reasoning is visible. Every tool page states what the tool is good at and where it falls down, in specific terms. If you disagree with a score, you should be able to read the page and see exactly which claim you are disagreeing with.
That is more useful than a number nobody can argue with, because a number nobody can argue with is usually a number nobody checked.
Benchmarks are separate, on purpose
Where a tool has published benchmark results, those live in their own section and stay untouched by our opinion of the tool. A benchmark is a fact about a test. A score is a judgement about usefulness. Blending them would let one launder the other.
You will occasionally see a tool with strong benchmarks and a middling score. That is not an error. It usually means the model is capable and the product around it is frustrating, and the product is what you would actually be using.
Why a score changes
Scores are not set once and archived. Tools get re-reviewed when they ship a major version, change pricing, lose a key capability, or fall behind a faster competitor.
A model that led its category in January can slip by summer without doing anything wrong, simply because something better arrived. The number should move when that happens rather than pretending the landscape stood still.
The reverse happens too, and more often than people expect. Tools that looked mediocre at launch have climbed ten points in a year by fixing the exact things we criticised.
How to actually use the number
Treat gaps under three points as a tie. Two tools scoring 88 and 90 are, for practical purposes, equally good, and the choice between them should come down to price, integrations, or which interface you can stand.
Treat gaps over ten points as real. That size of difference reflects a genuine capability gap that you will notice in the first hour.
And read the cons before you commit. The score compresses a lot of nuance into one number by design, and the thing that will actually annoy you is usually written out in full a little further down the page.
The three failure modes we watch for in ourselves
Any scoring system drifts. Ours has three specific ways of going wrong, and naming them is the only way we have found to catch them.
Recency bias is the first and worst. A tool reviewed the week it launched, riding a wave of enthusiastic coverage, tends to score higher than the same tool reviewed six months later doing exactly the same thing. We now re-review anything scored during a launch window, because launch-week judgement is not judgement, it is atmosphere.
Category inflation is the second. As a category matures, everything in it gets better, and if scores only ever move up the scale stops discriminating. When every image generator scores above ninety, the score has told you nothing you did not already know. So the top of a category gets recalibrated periodically, which means some tools drop points without getting worse. That is the system working, though it never feels that way to the tool that dropped.
The third is familiarity. Tools we use daily are easier to score generously, because we know their quirks and have stopped noticing them. Someone opening it for the first time has not made those accommodations. We try to weight the first hour heavily for exactly this reason.
What we get wrong
Plenty, and it is worth being specific rather than performing humility.
- We under-rate tools with bad marketing. A genuinely excellent product with a confusing website gets discovered late, and late discovery means a lower score for longer than it deserved.
- We over-rate breadth. A tool that does eight things adequately can score close to one that does a single thing brilliantly, and for most people the specialist is the better choice.
- We are slow on pricing changes. A tool that quietly doubled its price is scored on the old value for weeks until somebody notices.
- We have blind spots by language and region. Most of what we evaluate is English-first, and tools built primarily for other markets are under-represented.
None of these are solved. They are listed here because a scoring method that claims no weaknesses is either new or not being examined.
If you think a score is wrong
Tell us which claim is wrong rather than which number is wrong. The number is downstream of the reasoning, and the reasoning is what we can check.
The most useful corrections we receive follow the same shape: the page says this tool cannot do X, here is it doing X. That is actionable within a day. A message saying the score is too low is not, however strongly felt, because there is nothing in it to verify.
Corrections from the companies themselves are welcome on the same terms as corrections from anyone else. They get checked the same way and they do not get special weight, which is occasionally an awkward conversation and is the entire reason the score is worth reading.
See the scores for yourself
Browse every tool we have scored, sorted from highest to lowest.
Compare two tools side by side