You do not need a vector database for your first RAG chatbot

You want a chatbot on your website that answers questions from your own help pages. There are forty of them. You search for how to build it, and every tutorial starts the same way: convert your pages into embeddings, store them in a vector database, pay for both every month. Before you do any of that, read this. For forty pages — or four hundred — you probably do not need it.

What a RAG chatbot actually does

RAG stands for retrieval-augmented generation. The plain-English version: the AI model does not know your content. So before it answers, your system finds the few pages that match the question and pastes them into the prompt along with the question. The model reads those pages and answers from them.

That is the whole idea. Find, then answer. Everything else is detail. And the only real engineering question inside it is the first word: how do you find the right pages?

Two ways to find

Take a real question a customer types: “can I get a refund?”

Keyword scoring. Count how often the question’s words appear on each page. The refund policy page mentions “refund” six times, so it scores highest. The billing FAQ mentions it three times. The cancellation page twice. Send the top three to the model. This is the same maths that has powered search boxes since the 1990s. It is called BM25, it is about forty lines of code, and it runs on your own server in under a millisecond.

Embeddings. Turn every page into a long list of numbers that represents its meaning, do the same to the question, and find the pages whose numbers are closest. To do that you call an embedding API (your text goes to the provider), store the numbers in a vector database, and query it for every question.

Look at what embeddings buy you in that picture. One row. The “money-back guarantee” page does not contain the word “refund”, so keyword scoring misses it and embeddings find it. That is the advantage — pages that mean the same thing in different words. It is a real advantage. It is also a smaller one than the tutorials suggest, and there is a cheaper fix for most of it.

What the vector database costs you

Not just money, though there is that. Here is what you take on the day you add one:

• An API call for every page and every question. Your content and your customers’ questions leave your server to be turned into numbers. For a German or Dutch client, that sentence alone can end the project.
• A database to run or rent. One more service with its own password, its own outage, its own bill.
• Re-embedding when you edit. Change a page, and its numbers are stale until you regenerate them. Forget once, and the bot answers from last month’s policy.
• Debugging you cannot see. When keyword scoring picks the wrong page, you can print the scores and see why. When embeddings pick the wrong page, you get a similarity of 0.81 and a shrug.

None of these is fatal. All of them are work that a forty-page chatbot does not need.

Numbers from a system that runs this way

The assistant on this website answers from 136 chunks of text — service pages, prices, hiring terms, a few rules. Retrieval is BM25 with the standard settings (k1 of 1.5, b of 0.75), a small synonym list so “price” also matches “cost” and “rate”, and a title boost so a chunk whose heading matches the question scores higher. A typical question sends about 3,700 tokens to the model. Retrieval takes under a millisecond. There is no vector database, no embedding API, and nothing about your question leaves the server until the final call to the model.

One more thing that design gets right, and it matters more than the retrieval method: the rules do not compete for space.

Prices, hours and the rule “never promise a refund” sit in a pinned block that is sent with every request and never scored. Only the remaining 126 chunks compete for the rest of the budget. If you skip this, a rule can lose its place to an FAQ page on the very question where the rule mattered — the full story of that failure is here.

When you do need embeddings

Be honest about the crossover, because it exists:

• Thousands of documents. Past a few thousand chunks, keyword scoring starts returning too many near-ties and you want meaning to break them.
• Users who paraphrase heavily. Support for a product with lots of jargon, where customers describe the problem in their own words and never use your terms.
• More than one language. A German question against English pages will not match on keywords.
• You measured it. This is the real test. Log a hundred real questions from real users. For each one, look at which chunks BM25 returned. Count the misses. If it is under ten, you are done. If it is thirty, you have earned the vector database, and you will know exactly why.

Until you have that log, adding embeddings is guessing with a monthly bill attached.

The cheap fix for paraphrase

Most of the “different words, same meaning” problem in a small knowledge base is a synonym list. Twenty lines:

1const SYNONYMS = {
2  refund: ['money back', 'money-back', 'reimburse'],
3  price: ['cost', 'rate', 'fee', 'charge'],
4  start: ['onboard', 'begin', 'kick off'],
5}
6// Expand the question before scoring it.

Every time you find a missed question in your log, add a line. After a month the list stops growing, because your customers use a smaller vocabulary than you fear.

Start here

1. Split your pages into chunks of a few hundred words, one idea each.
2. Score with BM25. Any language has a library; it is also small enough to write yourself.
3. Put rules, prices and prohibitions in a pinned block that is always sent.
4. Log every question and the chunks it returned.
5. Add synonyms as the log shows misses.
6. Add embeddings only when the log proves the misses are about meaning, not words.

That order costs nothing to run, keeps the data on your own machine, and gives you a working assistant this week. The vector database will still be there when you need it, and by then you will be able to say why — which is the difference between building a system and following a tutorial. If you would like the version of this that runs on your own documents, that is what our generative AI work looks like, and the people who build it are here.