RAG & LLM Development

What does a RAG system need beyond embeddings and a vector search?

Five things demos skip: chunking tuned to the actual document type, hybrid search combining vectors with keywords, a reranking pass, permission filtering applied inside retrieval rather than after it, and an evaluation set. Generation quality is rarely the bottleneck. What reaches the prompt almost always is.

Chunking is where most quality is won or lost, and there is no universal setting — contracts, support tickets, API docs, and scanned PDFs all fragment differently. Hybrid search beats pure vector search on nearly every real corpus, because exact terms like product codes and error strings are precisely what dense embeddings blur. Add a reranker over the combined candidates and precision improves again. Permission filtering has to happen inside retrieval, not by discarding results afterwards, or you will eventually ground an answer in a document the user was never allowed to see.

RAG, fine-tuning, or a better prompt — which one do I need?

Prompting when the knowledge already sits in the model. Retrieval when it is private, changing, large, or must be cited. Fine-tuning when you need consistent behavior, format, or classification rather than new facts. The classic failure is fine-tuning to teach facts — slow, expensive, and stale on delivery.

Long context is the fourth option and often the right first move. If the corpus is small and static, putting it straight in the prompt beats building an ingestion pipeline you then maintain. Once it grows, changes often, or needs per-user access rules, retrieval becomes cheaper operationally. Combining approaches is legitimate but rarely the starting point: fine-tuning for output format while retrieval supplies facts is sound once both halves work individually, and impossible to debug before you have evaluations.

How do you keep LLM costs and hallucinations under control?

Costs are architectural: route each task to the cheapest model clearing its quality bar, cache aggressively, batch background work into pipelines instead of live calls, and cap tokens per request. Hallucinations are contained the same way — schema-validated outputs, grounding checks, citations, and an explicit path to refuse.

Most production workloads contain a few genuinely hard requests and a great many easy ones, and paying frontier prices for the easy ones is the single most common source of waste. Classification, extraction, and summarization often run well on smaller models. On containment, the useful mental model is that a language model will always produce an answer, so the system around it must be able to say no: structured outputs fail loudly instead of flowing downstream, grounding checks compare answers against retrieved sources, and returning nothing on low retrieval confidence beats any prompt instruction about not making things up.

Have you shipped a production RAG system?

One is running now: an Azure OpenAI “Dashboard Copilot” for a CLO compliance and portfolio SaaS, where a retrieval layer assembles approved database records into structured context and the model answers only from what it was handed. It is an ongoing engagement rather than a finished deployment, and the retrieval surface is deliberately narrow — in a compliance context, what may be retrieved at all is a governed decision.

The architecture is retrieval-augmented in substance rather than in branding. The copilot never runs the model against a free-floating document pile: a retrieval step pulls the records a given question is permitted to touch, a builder assembles them into a structured context payload, and generation is constrained to that payload — so an answer that cannot be grounded comes back as a refusal instead of a plausible number. Around it sit the parts that decide whether this survives a compliance review: guardrails on what may be asked, token budgets and cost controls per request, and a path from any answer back to the records that produced it.

Being precise about what that does not yet evidence. It is an ongoing engagement, so there is no multi-year production track record behind it. And because the corpus is approved structured data rather than an open document set, it does not demonstrate large-corpus chunking strategy, hybrid search tuning, or reranking at scale — those remain capability, built on the pipeline engineering documented alongside it.

That pipeline engineering is the half that transfers whether or not a retrieval step sits in the middle — ingestion, background jobs, retries, provider fallbacks, structured outputs, cost monitoring, observability. ScamMinder runs a 14-phase pipeline with a GPT-4o classification layer at sub-6-second analysis and 99.9% uptime across 20,000+ domains. The AI SEO platform orchestrates models through OpenRouter across 13 background pipelines with 74 test files behind it. And Continuity, the first M&A marketplace of its kind in the Belgian SME market, pairs a Claude-powered matching engine with an OCR dataroom handling sensitive deal documents across 5 roles and 6 access tiers in 3 languages — the same problem shape as permission-aware retrieval.