"Self-hosted is more cost-effective" is a claim I had been making for a while with nothing behind it. That is a preference wearing the costume of an engineering decision — and it reads as one: someone who cannot price the thing they chose did not choose it, they preferred it. So I measured it: a 27B model on one 24 GB GPU, generating at USD 0.43 of electricity per million tokens — against USD 5 for the cheapest hosted tier. That gap is real, and the rest of this page is the part nobody quotes: everything the 0.43 leaves out.
Everything I run for myself runs on my own hardware, and I can give you the architectural reasons in my sleep: no egress, no per-token bill, no dependency on somebody's pricing page. What I could not give you was the price. Not roughly. At all.
That is a problem beyond bookkeeping, because it makes a reasonable position unreadable. Someone hearing "I keep it local" with no figures attached has no way to tell an engineer who ran the numbers from an enthusiast who likes owning machines — and will, sensibly, assume the second. A trade-off you cannot quantify is indistinguishable from an ideology.
The obvious version of this benchmark compares two machines and is worthless: the delta is contaminated by everything else that differs between them. So the comparison is the same box, the same weights, the same prompts, the same meter — running on the GPU, then forced onto the CPU with options.num_gpu set to zero.
Tokens per second come from Ollama's own eval_count and eval_duration, so network round-trip and JSON parsing are not counted as inference. The prefill is kept separate rather than averaged in — it is a different cost.
Temperature zero, a fixed seed, a fixed token budget and a fixed prompt set. Without that, the generated length wanders between runs and moves tokens per second on its own, which looks exactly like a hardware difference.
The meter is sampled with the machine left alone, then throughout generation, then after. Both figures are kept, because the interesting number is the difference and you cannot get it from a single reading.
Every watt sample is written out with its timestamp, so someone who does not trust the figures I chose to print can recompute them. A published number without its working is an assertion with a table around it.
| Figure | What it is | When it is the honest one |
|---|---|---|
| Total | Every watt drawn while generating | You are deciding whether to own the machine at all |
| Marginal | Only the rise above idle | The machine is already on and you are choosing what to run on it |
Quoting whichever of the two supports the conclusion is the easiest way to lie with this data, so the report prints both, every time. For a desktop the marginal figure is the one to hold against a cloud price — because a desktop's socket is also measuring its monitors.
Electricity at USD 0.1583/kWh — the marginal rate off a dated bill, converted at the day's market rate, not the headline rate that includes the fixed charge. Cloud figures are output-token list prices, each with the URL and the day it was read.
| What | tok/s | Idle W | Load W | ΔW | Wh / 1M | USD / 1M |
|---|---|---|---|---|---|---|
| 27B on one 24 GB GPU | 11.3 | 29 | 140 | +111 | 2733 | 0.43 |
| same 27B, same box, forced to CPU | 1.2 | — | — | — | — | — |
| 7B on a CPU-only node elsewhere | 8.9 | — | — | — | — | — |
| hosted, small tier | — | — | — | — | — | 5.00 |
| hosted, mid tier | — | — | — | — | — | 15.00 |
| hosted, frontier tier | — | — | — | — | — | 25.00 |
num_gpu. That factor is measured. What is not measured is whether CPU-only is cheaper per token: it draws less peak power but takes 9.4x longer, and the rest of the machine keeps drawing the whole time. Which way that lands needs the wall meter — so this page does not claim it.One detail that decides the whole figure: the card runs on a 150 W cap, and under load it sits at 140 W — near the ceiling. An uncapped card would draw far more for a speed gain that does not scale with it. The cheap number is not a property of owning a GPU; it is a property of having limited one.
For a single operator, running and maintaining the machine is the largest cost by a distance, and none of it is priced here. A watts-only comparison flatters self-hosting by leaving out the person.
Already bought, not amortised into any of these figures. A number that ignores capital expenditure argues for owning the machine by leaving out the bill for the machine.
A node in someone else's datacentre has no plug to put a meter on. Those rows carry speed and no watts, and say so, rather than leaving a gap that reads like free.
This is throughput on a fixed prompt set. It says nothing about whether the smaller, cheaper model is good enough for the job — which is usually the question that actually decides it.