Polish Embedding Models - bge-m3, multilingual-e5 or PolDense? A Test on 900 Questions

An embedding model turns text into a vector of numbers, so RAG, a chatbot or a search engine can find passages similar in meaning to a question, not just ones with the same words. Which one should you pick for Polish text? The rule from my test on 900 real customer questions: if the Gemma licence works for you, take PolDense-150M (81.3 NDCG@10, 149M parameters, runs on an ordinary CPU). If you need an MIT or Apache licence, take multilingual-e5-large (80.7) or bge-m3 (79.7, the most robust to questions without Polish diacritics and with typos). Up to about 1M passages a CPU server is enough; above that you need a GPU or an API. Whatever you choose, add the query prefix the model requires - without it PolDense loses almost 30 points.

Which embedding model should you use for Polish RAG? I tested 10 models and BM25 on 900 real customer questions: results, robustness to missing diacritics and typos, the prefix that costs 30 points, the PIRB ranking, API prices, cost at two scales, licences and code.

In this post I show the results of my test of 10 models and BM25 on Polish customer questions, what happens when people type without Polish diacritics or with a typo, how one forgotten prefix ruins results, what the official Polish PIRB benchmark says, what API and local models cost at two scales, which licences allow commercial use and how to test models on your own data - with code.

Do You Need an Embedding Model at All - the First Decision

The simplest alternative is keyword search (BM25), which you get for free in Postgres or Elasticsearch. In my test BM25 scored 47.7 NDCG@10, and the best embedding model 82.7. The correct passage came first in 33% of cases with BM25 and in 67% with the best model. The reason is simple: a customer asks "how do I send the item back", the documentation says "return procedure", and Polish inflection means "zwrot", "zwrotu" and "zwrocie" are three different words to BM25.

My Test: 10 Models on 900 Customer Questions

How I tested: the Allegro FAQ set from the PolEval 2022 competition - 900 real customer questions and 921 answer passages (50 words on average). The correct passages are known for each question. Each model turned all passages and questions into vectors, and I checked whether the right passage made it to the top of the list. Everything ran on a 4-core Intel Xeon CPU, no GPU, with the prefixes recommended by the model authors (October 2026).

What follows from this:

• Polish models win in their weight class. PolDense-68M, with 68M parameters, beats multilingual models eight times its size (mmlw-large, Qwen3-0.6B) and almost matches bge-m3.
• Popular doesn't mean good in Polish. Qwen3-Embedding-0.6B, often recommended in international rankings, came second to last among the models - on par with the smallest e5.
• Parameter count says nothing about speed. PolDense-68M is smaller than mmlw-base but slower on a CPU, because it handles texts up to 8,192 tokens and without special acceleration libraries (available mainly on GPUs) it computes more slowly.
• The gap between first and fifth place is 3.6 points. The choice between them should come from licence, speed and robustness to messy questions, not the ranking alone.

Polish Diacritics and Typos - the Test Nobody Runs

Customers often type "jak zwrocic towar" instead of "jak zwrócić towar". So I checked what happens when I strip Polish diacritics from the same questions (92% of them were affected) or add one typo to a longer word:

Conclusions: bge-m3 is the most robust - with questions lacking diacritics and with typos it loses the least, so for consumer-facing search it can be a safer choice than a model with a higher score on clean data. BM25 and the smallest models suffer most. Don't strip Polish diacritics from documents "just in case" - the models were trained on correct Polish and you'll only make results worse.

The Prefix Everyone Forgets

Many models require a short prefix before the question: "query: " in e5, "zapytanie: " in mmlw, "[query]: " in PolDense. It's not cosmetic. I measured what happens when you leave it out:

Without the prefix, the best model in the test drops almost to the level of plain keyword search. This error doesn't crash the application or show a message - results are just worse and nobody knows why. So keep the prefix in one place in the code together with the model name, and add it only to questions, not documents (unless the model also requires a document prefix, like e5 with "passage: ").

from sentence_transformers import SentenceTransformerMODEL = "OPI-PIB/PolDense-150M"QUERY_PREFIX = "[query]: "  # changing the model? change the prefix toomodel = SentenceTransformer(MODEL, device="cpu")def embed_documents(texts: list[str]):    return model.encode(texts, batch_size=16, normalize_embeddings=True)def embed_query(text: str):    return model.encode(QUERY_PREFIX + text, normalize_embeddings=True)docs = ["Zwrot towaru możesz zgłosić w ciągu 14 dni w zakładce Moje zakupy.",        "Fakturę VAT pobierzesz po zaksięgowaniu płatności."]D = embed_documents(docs)q = embed_query("jak oddac towar ktory mi nie pasuje")print(docs[int((D @ q).argmax())])

The PIRB Ranking - the Bigger Picture Across 41 Tasks

My test covers one type of data: short FAQ answers. A broader view comes from the Polish Information Retrieval Benchmark (PIRB), maintained by Poland's National Information Processing Institute (OPI): 41 tasks, from law and medicine to questions from Polish web portals. Selected results (average NDCG@10, as of 9 October 2026):

The order in PIRB is similar to mine: PolDense models at the top, e5-large ahead of bge-m3, and Qwen3-0.6B far behind. The difference is the scale of the scores - PIRB includes many harder tasks (e.g. law), so averages are lower. A curiosity from PIRB: the hybrid variant of bge-m3 (55.75) scored slightly lower than the dense variant alone (55.98), so "hybrid always helps" is a myth - you have to check it on your own data.

API Models - OpenAI, Gemini, Voyage

API models aren't in PIRB, and without keys I couldn't test them fairly, so I don't give Polish results for them. Prices (October 2026):

An API makes sense when you don't want to run a server and the data can go to the provider under business terms. Before choosing, run them through the same test I show below - testing on 1,000 passages costs fractions of a cent.

What It Costs at Two Scales

Assumption: passages of about 300 tokens and questions of about 20 tokens.

The key conclusion: computing embeddings through an API is cheap. The real costs are elsewhere:

• Vector database memory. 10M vectors with 3072 dimensions (text-embedding-3-large) take about 123 GB, with 512 dimensions (PolDense-68M) about 20 GB. That's a difference in the database bill every month. More on database costs in the vector database comparison.
• Switching models. Vectors from different models aren't compatible, so a model switch means recomputing the whole index. Keep the source texts and plan for it up front - I cover this in the post on AI vendor lock-in.
• Privacy. A local model sends documents nowhere, which simplifies GDPR compliance for personal data.

Licences - What's Allowed in a Company

If your legal team prefers the simplest licence, pick e5-large or bge-m3 - you lose 1-2 points against PolDense-150M, but you get MIT.

How to Test Models on Your Own Data

A ranking is a starting point, not a decision. Collect 50-200 real questions and mark for each the passage that should be found. Then run a script like this for 2-3 candidates:

from sentence_transformers import SentenceTransformerCANDIDATES = {"OPI-PIB/PolDense-150M": "[query]: ", "BAAI/bge-m3": "",              "intfloat/multilingual-e5-large": "query: "}DOC_PREFIX = {"intfloat/multilingual-e5-large": "passage: "}def evaluate(name, qprefix, docs, questions, correct):    m = SentenceTransformer(name, device="cpu")    D = m.encode([DOC_PREFIX.get(name, "") + d for d in docs], normalize_embeddings=True)    Q = m.encode([qprefix + q for q in questions], normalize_embeddings=True)    top10 = (Q @ D.T).argsort(axis=1)[:, ::-1][:, :10]    hit1 = sum(top[0] == c for top, c in zip(top10, correct)) / len(correct)    hit10 = sum(c in top for top, c in zip(top10, correct)) / len(correct)    return round(100 * hit1, 1), round(100 * hit10, 1)# docs: list of passages, questions: list of questions, correct: index of the right passage for each question# for name, prefix in CANDIDATES.items(): print(name, evaluate(name, prefix, docs, questions, correct))

Add questions without Polish diacritics and with typos to the set - the way your users actually type. The "correct passage in top 10" score tells you whether RAG has a chance to answer well, because the language model usually gets several top passages, not just the first.

The Most Common Mistakes

• No query prefix - up to 30 points lost in my test, with no error message at all.
• Choosing a model from an English ranking. MTEB and international rankings don't tell you how a model handles Polish.
• Stripping Polish diacritics from documents. The models were trained on correct text, so you only make results worse.
• Mixing vectors from two models in one index. After a model switch, recompute everything.
• Passages too long for 512-token models. e5 and mmlw truncate longer text without warning, so the end of a document never reaches the vector. Split texts into shorter passages or choose a model with a longer limit (bge-m3, PolDense - up to 8,192 tokens).

---

I help companies build search and RAG on Polish documents - from choosing an embedding model and testing it on your own data to the vector database and deployment. I do this as part of AI app development and AI automation. Get in touch - I'll start by testing 2-3 models on a sample of your questions.

Worth reading next:

• Vector Databases - pgvector, Pinecone, Qdrant or Weaviate? How to Choose for RAG and AI Memory
• Advanced RAG - Chunking, Hybrid Search and Reranking
• How to Build an Internal Company Knowledge Base with AI (RAG in Practice)
• AI Agent Memory - Mem0, Zep, Letta or LangMem?