Comparison · · 6 min read
Open weights or closed API. How to actually decide
The gap in quality has nearly closed. The gap in what you take on has not. A straight comparison of what each choice costs you.
By Manish
Two years ago this decision was easy. Closed models were substantially better, so you used a closed model and the conversation ended. Open weights have since landed within a couple of points of the frontier on most published benchmarks, and the question got genuinely difficult.
It is no longer really a quality decision. It is a decision about what you are willing to operate, and that is a different kind of question with a different kind of answer for every team.
What a closed API actually buys you
The value is mostly in what you do not have to think about, which makes it easy to undervalue until you try the alternative.
- Somebody else owns uptime, scaling, and the GPU supply problem.
- You get the newest model the week it ships, with no migration project.
- Safety tuning, abuse handling and content filtering come included and maintained.
- The engineering you do not do is engineering you can spend on your actual product.
What you give up is control, in four specific ways. Pricing can change with notice you did not agree to. Models get deprecated on the provider's schedule rather than yours. Your data leaves your network on every call. And rate limits can bind exactly when you are succeeding.
What open weights actually buy you
The value is control, and control is worth different amounts to different organisations.
- The data never leaves your infrastructure, which settles most compliance conversations before they begin.
- The model cannot be deprecated out from under you. The weights are on your disk and they will still work in five years.
- At high volume the economics invert, sometimes dramatically.
- You can fine-tune deeply, inspect behaviour, and modify things a hosted API will never expose.
What you take on is everything else: serving, scaling, monitoring, evaluation, security patching, and the person whose phone rings at two in the morning. That person is the real cost and they rarely appear in the comparison spreadsheet.
Open weights are not free. They move the cost from your card statement to your engineering calendar, where it is harder to see and harder to cancel.
The quality gap, honestly
On general benchmarks the best open-weight models now sit within roughly two points of closed leaders. That gap is smaller than the gap between closed leaders themselves, and small enough to be within run-to-run variance on several tests.
The gap is wider in specific places, and knowing where matters more than the average. Frontier reasoning on genuinely hard problems still favours closed models. Very long context handling favours them too. Tool use and structured output are closer than they were but still generally better on the closed side, largely because those capabilities get product attention rather than research attention.
For the large middle ground of summarising, drafting, classifying, extracting and answering questions over your own data, the gap has effectively closed. Most production workloads live in that middle ground.
The cost crossover
The arithmetic is not subtle once you write it down. A closed API costs per token with no fixed cost. Self-hosting costs a fixed amount for GPUs whether you use them or not, plus the engineering time.
Below a certain volume the API is obviously cheaper, because idle GPUs are pure waste. Above it, self-hosting wins and keeps winning as you scale. Where that crossover sits depends on your traffic shape, and bursty traffic pushes it much higher than steady traffic, because you must provision for the peak.
Two things people consistently get wrong here. They forget that self-hosting needs headroom for spikes, so you are paying for capacity you mostly do not use. And they omit engineering salary, which for a small team is frequently larger than the entire inference bill either way.
Compliance is where this gets decided
For a lot of organisations the technical comparison never happens, because a regulatory requirement settles it first.
If you handle health records, legal documents under privilege, or data with a hard residency requirement, self-hosting may be the only route that passes review. Zero-retention API agreements have improved this substantially and cover many cases, but not all, and not in every jurisdiction.
Worth checking properly rather than assuming. Several teams have self-hosted at considerable expense to satisfy a requirement that a zero-retention agreement would have met, and several others have shipped on an API in a situation that genuinely required otherwise.
The answer most teams arrive at
Both, routed. A closed frontier model for the hard reasoning that justifies the price, an open model for the high-volume work that does not.
Routing between them used to be a project and is now largely a configuration. Gateways that expose one interface across many providers make this straightforward, and the savings are frequently large, because in most applications a small fraction of requests genuinely need the expensive model.
This also hedges the risk that made people nervous in the first place. If a provider raises prices or deprecates a model, you already have a second path in production rather than a migration to plan.
How to decide, in order
Start with compliance, because it can end the discussion. Then look at volume, because below the crossover the API wins on cost and nothing else will change that. Then ask whether you have someone who genuinely wants to own inference, and be honest, because a reluctant owner is worse than no owner.
If you are still finding product-market fit, use a closed API and stop thinking about it. Iteration speed beats cost per token when you do not yet know what you are building. Optimise inference when inference is a meaningful line item, not before.
Fine-tuning is less of an argument than it used to be
The ability to fine-tune deeply is often cited as the decisive advantage of open weights. It was a stronger argument two years ago than it is now, and it is worth updating.
Most teams who thought they needed fine-tuning actually needed better prompting or better retrieval. Models improved enough at following instructions that behaviour previously requiring a tuned model is now achievable with a well-written system prompt and good examples.
Fine-tuning still wins clearly in three situations: a genuinely specialised domain vocabulary the base model handles poorly, a strict output format that must never vary, and high-volume tasks where a small tuned model can replace a large general one at a fraction of the cost.
Outside those, it is frequently an expensive detour that also locks you to a base model version. Hosted fine-tuning on closed models exists too, which narrows the gap further.
The operational reality of self-hosting
Worth spelling out, because the comparison usually skips it and it is the part that decides whether this works.
- GPU supply and cost. You provision for peak, so you pay for capacity that is idle most of the time.
- Serving infrastructure. Batching, queuing and load balancing are solved problems with mature tooling, and someone still has to run them.
- Monitoring. Latency, throughput, error rates and output quality, which is harder to monitor than the others.
- Upgrades. New open models ship constantly, and evaluating and migrating is recurring work rather than a one-off.
- On-call. Somebody owns it when it breaks at an inconvenient hour.
None of this is exotic and all of it is a real job. The teams for whom self-hosting works well usually have platform engineers already. The teams for whom it goes badly assigned it to a product engineer as a side project.
Local models are a third option people forget
For individual use rather than production serving, running a model on your own machine has become genuinely practical, and it sidesteps the whole debate.
A capable laptop now runs models that would have been frontier-class two years ago. Cost per use is zero, the data never moves, and it works without a network. For summarising documents, drafting, and code assistance on a personal machine, this is frequently sufficient.
It does not scale to a product, and it is not meant to. But a surprising number of people paying for an API are doing work a local model would handle, and the setup is now a single command rather than a weekend.
Compare the models before you choose
See open and closed models ranked side by side, with what each one costs.
Browse AI tools for building apps