Self-Hosted LLM: When Does Running Your Own Model Beat an API?

A self-hosted LLM beats a hosted API in two cases: when your prompts hold data that must not leave servers you control, or when you send so many requests that a server running all month costs less than paying per token. On price alone it seldom wins against a small hosted model. A cloud server with one 16 GB graphics card costs about US$384 a month at list price before it answers a single question. The main reason to host your own model is where the data goes.

What is a self-hosted LLM, and what do you run?

A self-hosted LLM is an open-weight language model that runs on a machine you control: a server in your own cloud account, a rented GPU server or a box in your office. You download the model's weights once. An inference server loads them and answers requests over your own network. Nothing you send reaches a model company, because no model company is in the path.

The setup has three parts:

• The model. An open-weight model such as Qwen3, Mistral Small, Llama 3.1, Gemma 3 or gpt-oss. Bigger models usually write better and need more memory.
• The inference server. Software that loads the model and answers requests. llama.cpp runs on almost anything, from a laptop to a GPU server, and Ollama is a quick way to get models running on one machine.
• The machine. A GPU server for real traffic. A laptop is enough to test whether a model can do your job at all.

Inference servers such as llama.cpp answer in the same request format as the OpenAI API, so an app that already calls a hosted model often needs only a new address and a new model name. With llama.cpp installed, this line downloads a small model and starts that kind of server on your own machine:

1llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Which models can a business run on its own server?

Start with the licence, because it decides whether you may use the model in a product at all. These are the licences the model cards on Hugging Face named on 2 November 2026:

Apache 2.0 is a standard open licence that allows commercial use. The others come with their publisher's own terms. Read them, and keep a copy with your project, before you build a product on the model.

How much does a self-hosted LLM cost compared with an API?

A self-hosted LLM costs the same for every hour the server is on, busy or idle. An API costs nothing when nobody asks and a small amount for every request. So the answer depends on how many requests you send. Take one example request: 1,500 tokens in (a question plus a page of your own documents) and 300 tokens out (a short answer), priced from list prices read on 2 November 2026. Swap in your own numbers.

What does the server cost each month?

Two single-GPU servers from Microsoft's public price list, in its East US region, on Linux, paid as you go:

The same price list has lower spot prices too. The monthly figure is what you pay with the server switched on all the time, which a live chat feature needs.

What does the same work cost on an API?

Anthropic's price page lists these prices per million tokens:

Where do the two bills cross?

Divide the server's month by the price of one request. The result is the number of requests at which the server becomes the cheaper option:

The second row looks tempting. It holds only if a model that fits on a 16 GB card does your job as well as Sonnet does, and on hard writing and reasoning work it usually does not. Test it on your own task. Run the same 50 real requests through the open model and the hosted one, and compare the answers before you compare the bills.

What does running the server cost in people's time?

Someone has to run the server. They apply security updates, watch the queue, move to a newer model when one comes out and fix the server when it stops at night. That is engineering time every month. Ten hours a month at the published AI rate on the Boffin Coders rate card, US$25-40 an hour, adds US$250 to US$400, about the price of the T4 server again.

What hardware does a self-hosted LLM need?

Memory decides what fits. A model's weights take its number of parameters times the bytes each parameter is stored in. At 16-bit precision each parameter takes 2 bytes. Quantisation stores them in fewer bits. llama.cpp supports 1.5-bit up to 8-bit, which cuts the memory a model needs and speeds up its answers. Check the answers again at the setting you pick.

That is the weights alone. The model also needs room for the text it is working on, and that grows with longer prompts and with more people asking at once. So leave space on the card. An 8 billion parameter model at 4-bit runs on the 16 GB T4 with room to spare, while the same model at 16-bit would fill it. A 70 billion parameter model at 4-bit fits on the 80 GB A100, and at 16-bit it needs two of them.

Can you test a model on a laptop before you buy a server?

Yes, and it is the cheapest step in the whole project. llama.cpp treats Apple silicon as a first-class target and runs on ordinary processors too, so a laptop with 16 GB of memory can run a small model at 4-bit. It will be slower than a GPU server, and that is fine here: the test checks whether the answers are good enough on your own tasks.

Is an on-premise LLM more private than an API?

An on-premise LLM runs on machines in your own building. A self-hosted one can also sit in your own cloud account. Either way, your prompts, your documents and the model's answers stay on machines you control, and no model company receives a copy. Hosted APIs publish their own terms on how they store and use what you send, but the data still leaves your network to reach them.

The privacy work moves to you with the server. You decide who can call the model, what gets logged and for how long, and who can read the backups. A self-hosted model with an open port and every prompt logged forever is less private than a well-run API. Put the server on a private network, keep logs short and write down where every prompt goes, so you can answer a client who asks.

Which part of your system should you host yourself?

Find the step where the sensitive data first enters your system, and host the model that works on it. That is often a small model that is cheap to run, while the LLM further along can stay hosted.

On an exam platform we built for a coaching institute, the speech-to-text model runs on the institute's own server, on an ordinary processor with no graphics card, one recording at a time, and no recording goes to a model vendor. The scoring step after it sends only the text to a hosted model, and a teacher approves every grade it drafts. The speech-to-text pipeline write-up shows the code and the trade-offs, including that a small model on a processor is less accurate on strong accents than the best hosted one.

The same split works for many businesses. A small model on your own server reads the contract or the customer file and takes out the names and numbers, and the hosted model only sees what is left.

When is a hosted API still the better buy?

• Your volume is low or comes in bursts, so a server would sit idle most of the month.
• You need the best quality on hard writing, coding or reasoning, and no model that fits your budget matches it.
• Nobody on your team will look after a server at three in the morning.
• You are still finding out whether the feature is worth building at all.

In those cases start on the API, and keep your prompts in a format any OpenAI-style server can answer, so moving later stays a small job.

How do you host your own LLM, step by step?

To host your own LLM without spending a month on the wrong model, work through the steps in this order:

1. Write down the job. Pick one task, such as summarising support tickets or answering from your own documents, with 50 real examples and the answers you would accept.
2. Test open models on a laptop. Run two or three models from the licence table through your 50 examples and keep the smallest one that passes.
3. Size the server from the memory sum. Add the weights at your chosen precision, the room for the text in flight and the number of people asking at once.
4. Serve it on a private network. Run the inference server with no public port, put your own API in front of it and point your app at its address.
5. Log what you need and nothing more. Record request counts, response times and errors, and keep prompt text out of the logs unless you have a reason to keep it.
6. Compare a month of bills. Put the server, the hours spent on it and the hosted API price for the same traffic side by side, then decide whether to keep it.

If you would rather have a private LLM deployment planned, built and looked after for you, that is work we take on, and we invoice after you have seen it running.