RAG Implementation for Business – AI Knowledge Base & Vector Search

Do you need custom RAG – long context, an off-the-shelf Copilot or a bespoke system

Custom RAG makes sense when your documents don't fit in one context window, when different people should see different documents and when answers must point to their source. If the knowledge fits in a few hundred pages, or you already run a tool with well-managed permissions, something simpler often does the job.

In its write-up of Contextual Retrieval (2024), Anthropic says it plainly: if your knowledge base is under 200,000 tokens (about 500 pages), you can skip RAG and put the whole thing in the prompt, and prompt caching then cuts cost by up to 90% and latency by more than half. Beyond that size the problem isn't just the limit. The “Lost in the Middle” study (Liu et al., TACL 2024) found that models use information at the start and end of a long context best and the middle worst. Chroma's “Context Rot” report (2025) tested 18 models and confirmed that reliability drops as input grows, even on simple tasks. With thousands of documents, giving the model a few relevant passages beats handing it the whole archive.

Off-the-shelf assistants inherit whatever permissions you already have. Microsoft's Copilot readiness documentation (2026) says Copilot and agents retrieve data from Microsoft Graph while respecting existing permissions and sharing settings, and also notes that SharePoint defaults to the most permissive sharing option. So if the payroll folder was shared with “everyone in the company”, the assistant will surface it. Tidy permissions are a precondition for every one of these paths.

Preparing documents for RAG – inventory, parsing and metadata

RAG quality starts before indexing: the system can only answer as well as it reads your documents. The biggest problems I see are scans with no text layer, tables broken into loose words and several versions of the same procedure with nothing saying which one applies.

Inventory checklist

• Sources: where documents live (drive, SharePoint, Confluence, Notion, helpdesk, CRM) and whether they can be read via API or only by export.
• Owner: who keeps each source current. No owner, no place in the index.
• Versions: which version is in force; I remove old versions or tag them in metadata so they don't win in search.
• Format: text PDF, scan needing OCR, DOCX, spreadsheet, wiki page, email. Each needs a different parser.
• Permissions: who may see each document; I store this alongside every chunk in the index.
• Personal data: whether documents contain customer or employee data and whether it belongs in a knowledge base at all.

Document type decides how it's processed

Every chunk gets metadata: source, title, section, effective date, language and permission group. That lets you filter retrieval (say, only this year's sales procedures) and show the user where an answer came from. A production example on unusual documents is the quoting engine in the Konkret case study: 70 historical cost estimates totalling 3.2 GB won't fit in any context window, so the model gets only the few most similar estimates picked by retrieval.

Chunking and indexing – where I start and what I change after testing

I start by splitting text into chunks of a few hundred to around a thousand-plus characters with a small overlap, then change the strategy for document types where tests show errors. No single chunk size suits contracts, manuals and FAQs at once.

In my Research Agent demo on this site, the chunker packs whole sentences up to 1,200 characters and carries 180 characters of overlap into the next chunk, so a sentence on the boundary keeps its context. That's a reasonable baseline for running text. In production I add three things the demo deliberately leaves out: hybrid search (vectors plus BM25), reranking and context prepended to each chunk.

Chunk context is a short line such as “From the service agreement with Company X, clause 7, penalties”, generated by a model at indexing time. Anthropic (2024) put the one-off cost at about $1.02 per million document tokens with prompt caching. For a typical company knowledge base that's negligible next to the time people lose on bad answers. Chunking, hybrid search and reranking with code are covered in Advanced RAG – chunking, hybrid search and reranking.

What I check in the index before the first quality test

• A random 30–50 chunks: do they make sense without the rest of the document, and do they cut off mid-list or mid-table?
• Did numbers, dates and product names survive parsing intact?
• Does every chunk carry full metadata, including its permission group?
• Does updating a document replace its chunks rather than add new ones next to the old?
• Does deleting a document at the source remove it from the index?

The choice of vector database rarely decides quality. It decides cost, hosting and whether hybrid search and metadata filtering come built in. I compare pgvector, Pinecone, Qdrant and Weaviate in Vector databases – how to choose for RAG.

RAG beyond English – embeddings, inflection and hybrid search

Inflected languages such as Polish make retrieval harder, so for non-English documents I almost always combine vector search with keyword search and test the embedding model on the company's own questions rather than on a public leaderboard.

For Polish the best public reference is PIRB (Polish Information Retrieval Benchmark), built at Poland's National Information Processing Institute and published at LREC-COLING 2024. It spans 41 retrieval tasks, including medical, legal and business sets. The authors compared more than 20 models and concluded that dense models perform best and hybrid methods improve results further. That matches what I see in projects: semantic search alone misses contract numbers, product codes and proper names, while BM25 alone doesn't see that “terminating the contract” and “giving notice” are the same question.

How I choose an embedding model

1. Collect 50–100 real questions from staff or customers, including typos, abbreviations and industry jargon.
2. Map each question to the document and chunk that should be found.
3. Compare 2–4 models (commercial and open, multilingual and language-specific) on the same index, measuring how often the right chunk lands in the top few results.
4. Only then weigh cost, vector size and whether the model can run on-premises.

Switching embedding models after launch means re-indexing the whole base, because vectors from different models aren't comparable. That's why I settle it during testing, not after the first user complaints.

RAG permissions and security – who sees what, and what a document can inject

In RAG, permissions must be enforced at retrieval time, not when the answer is displayed: a chunk the user may not see must never reach the model's context. The second risk is documents carrying hidden instructions for the model.

OWASP's Top 10 for LLM Applications 2025 added a dedicated category, LLM08: Vector and Embedding Weaknesses, aimed squarely at RAG systems. It lists data leaks through misaligned access controls, cross-tenant leakage in a shared vector store, knowledge-base poisoning and recovery of source text from the vectors themselves. Its example is a CV with hidden text instructing the system to recommend the candidate.

Rules I apply in every RAG build

• Permission filter in the query: every search carries the user's groups, taken from the login system, never from the question text.
• Permission sync: when someone loses access to a folder at the source, they lose it in the assistant at the next sync.
• Separate collections for external users: a customer-facing assistant never shares a collection with the internal one.
• Content checks on ingest: I strip hidden text, white-on-white fonts and comments, and treat outside documents (CVs, bids, emails) as untrusted.
• Chunks are data, not commands: the prompt separates instructions from document content, and the model has no tools a document could trigger.
• Retrieval logs: I record who asked what and which chunks they got, so every answer can be explained.

Vectors are data too. Since text can be partly reconstructed from them, a vector store needs the same protection as a document store: encryption, backups and access control. Attacks on RAG and agents are covered in Prompt injection and LLM application security, and personal data in prompts in GDPR and AI.

RAG evaluation – how I measure retrieval and answer quality before launch

I measure RAG quality separately for retrieval and for answers, because they're two different sources of error. If the right chunk wasn't found, no model will answer well; if it was found and the answer is still wrong, the problem is in the prompt or the model.

RAG doesn't guarantee zero hallucinations. A Stanford team (Stanford HAI, 2024) tested commercial RAG-based legal research tools: Lexis+ AI gave incorrect information in more than 17% of answers and Westlaw AI-Assisted Research in more than 34%. These were tools from major publishers with strong databases. So I treat a “zero hallucinations” promise as a red flag and give clients a measured error rate on their own questions instead.

What goes into the test set

Metrics and thresholds

• Retrieval: whether the right chunk is in the top k results (recall@k) and how high it ranks. This runs without the generating model, so it's fast and cheap.
• Answer: faithfulness to sources (every claim follows from the chunks), relevance to the question and citation accuracy. The Ragas library calls these Faithfulness, Response Relevancy, Context Precision and Context Recall, among others.
• Refusals: the share of correct “I don't know” answers on unanswerable questions, and the share of unnecessary refusals where an answer exists.
• Production: response time, cost per question, share of answers rated negatively by users.

I agree acceptance thresholds with the client before building, not after the first results. Part of the scoring is done by a model acting as judge, but I calibrate it by hand on a sample, because judges get things wrong too. The method is described in LLM evaluation with LLM-as-a-judge, and wiring tests into the release process in How to write tests for AI applications.

Classic RAG, agentic RAG and GraphRAG – which variant to choose

For most company knowledge bases, classic RAG with hybrid search and reranking is enough. I add agentic RAG when questions are complex and need several searches, and consider GraphRAG only for questions about a whole corpus, such as main themes or relationships.

GraphRAG was described by Microsoft Research in “From Local to Global” (2024). The authors show that plain RAG struggles with questions like “what are the main themes in this dataset?”, because that's summarising a corpus rather than retrieving one passage. Indexing, however, runs the whole corpus through a model, so it suits stable, high-value collections, not a wiki that changes daily.

I show agentic RAG in the Research Agent demo. One lesson from building it applies to every project: the first step forces a knowledge-base search. Without it, the model sometimes answered straight from the question and, following the “answer only from the base” instruction, refused questions the base actually covered. Later steps are left to the model, and in the final step tools are switched off so an answer always comes out. Designing these loops is covered in Context engineering. When the assistant should also take actions in your systems, that's the scope of AI agents for business.

RAG implementation cost and running costs – what they're made of

My RAG implementations start from €1,400, and running costs come from four items: embeddings, the vector database, the model that writes answers, and maintenance. Embeddings are usually the cheapest part; the answering model and people's time keeping documents current cost the most.

A worked example: OpenAI's pricing (2026) lists text-embedding-3-small at $0.02 per million tokens. If I assume a page of a company document is about 500 tokens, indexing 10,000 pages (5M tokens) costs around $0.10, and a few dollars with chunk context at Anthropic's estimate. That's why RAG pricing is about answer cost and document work, not vectors. Implementation price depends mainly on the number of sources to connect, document quality (scans, tables), permission requirements and hosting: cloud or self-hosted. Local models are covered in Ollama, vLLM and self-hosted LLM.

Maintaining a RAG system after launch – updates, data deletion and monitoring

A RAG system needs maintenance because documents, user questions and models all change. Without monitoring, quality slips quietly: the new price list never reaches the index, or the old procedure keeps beating the new one.

What I monitor

• Unanswered questions: the list of questions where the system said “I don't know”. It's the best signal of what knowledge is missing from your documents.
• User ratings: I review negatively rated answers weekly in the first month, then monthly.
• Index freshness: time from a document change at the source to its update in the index, plus sync errors.
• Test set scores: after every change to the model, prompt or chunking.
• Cost and latency: daily model spend and response time, with an alert on spikes.

RAG beats training a model on one important point: deleting data. When a customer or employee asks for their personal data to be erased, I remove the specific chunks from the index and it's done. You can't pull it out of a trained model's weights without retraining, which I cover on the LLM fine-tuning page. Metrics, tracing and alerts for LLM apps are covered in Monitoring AI in production.

What to prepare before a RAG implementation – a checklist for your team

Projects move fastest when the company names its sources, document owners and a few dozen real questions before kick-off. That shortens the inventory phase and lets me build the test set straight away.

1. A list of sources with access (a service account or export) and a note on which ones are current.
2. A person on your side who answers questions about document content and signs off test results.
3. 50–100 questions people actually ask, ideally taken from email, the helpdesk or chat, with where the answer lives.
4. Access rules: which employee groups see which documents.
5. A hosting decision: cloud model under a data processing agreement, or everything on your own infrastructure.
6. Where the assistant will live: browser chat, Slack, Teams, intranet or a customer widget.

When the assistant serves customers rather than staff, RAG gets joined by order data, handover to a human and tone of voice; that's the scope of AI chatbot for business. The bigger picture, including process selection, costs and GDPR in automation, is on the AI & Automation page, and a practical walkthrough of building a knowledge base is in How to build a company internal knowledge base with AI.