RAG Pipeline: Retrieval Augmented Generation, Step by Step

Retrieval Augmented Generation is a plain idea wearing an intimidating name. Before you ask the model a question, go and find the handful of documents that are relevant, paste them into the prompt, and ask the question with that context attached. That is the whole thing.

The idea takes an afternoon. Building a RAG chatbot that returns correct answers takes considerably longer, and almost every failure turns out to be in retrieval rather than in the model. This is the pipeline end to end, with the parts that actually break called out.

Why RAG rather than a bigger prompt

With million-token context windows, "just put everything in the prompt" is a real option, and for a small corpus it is the right one. If your knowledge base is twenty pages, skip this entire post, paste the twenty pages, and turn on prompt caching.

Retrieval starts earning its complexity when the corpus does not fit, when it changes often enough that a cached prompt would be stale, or when you need to show a user which source an answer came from. That last one matters more than people expect. An answer with a citation gets trusted and checked; an answer without one gets trusted and not checked.

The pipeline, in two halves

Everything in RAG happens in one of two phases, and keeping them separate in your head prevents a lot of confusion.

Indexing is a batch job you can afford to make slow and careful. The query path is on the critical path of a user waiting for an answer, so every millisecond there is visible.

Step 1: chunking

You cannot embed a 90 page PDF as one vector and get anything useful. The document has to be split, and how you split it determines the ceiling on retrieval quality for everything downstream.

type Chunk = {
  id: string;
  text: string;
  documentId: string;
  heading?: string;
};

// Naive fixed-size chunking with overlap. Fine as a starting point.
function chunkText(text: string, size = 1000, overlap = 200): string[] {
  const chunks: string[] = [];
  for (let i = 0; i < text.length; i += size - overlap) {
    chunks.push(text.slice(i, i + size));
  }
  return chunks;
}

The overlap exists so a sentence that straddles a boundary still appears whole in one of the two chunks. Without it, the single most relevant sentence in your corpus can be cut in half and retrieved by nothing.

Fixed-size splitting is a starting point and not a good one. It cuts through the middle of tables, separates a heading from the paragraph it introduces, and produces chunks that begin mid-clause. Splitting on structure instead, headings, then paragraphs, then sentences, is a larger improvement to answer quality than almost anything you can do to the prompt afterwards. I went into the strategies and how to measure them in chunking and embeddings.

Step 2: embeddings

An embedding turns text into a list of numbers positioned so that text with similar meaning lands nearby. That is what lets a question about "cancelling my subscription" find a document titled "ending your plan", which no keyword search would return.

// Embed in batches. One HTTP call per chunk is slow and usually rate limited.
async function embedAll(texts: string[]): Promise<number[][]> {
  const vectors: number[][] = [];
  const batchSize = 96;

  for (let i = 0; i < texts.length; i += batchSize) {
    const batch = texts.slice(i, i + batchSize);
    vectors.push(...(await embed(batch)));
  }
  return vectors;
}

Step 3: storing and searching

A vector database stores vectors with their text and metadata, and answers nearest-neighbour queries fast. Qdrant, pgvector, Pinecone, Weaviate and several others all do this competently.

My recommendation for most projects is Postgres with the pgvector extension, and it is a boring one on purpose. Below a few million chunks the performance difference is not what will limit you, and having your vectors in the same transaction as your application data removes an entire category of "the index and the database disagree" bugs. Reach for a dedicated vector database when scale or a specific feature actually demands it.

// Store the text next to the vector. You need it to build the prompt,
// and a second lookup to fetch it is a wasted round trip.
await db.query(
  `INSERT INTO chunks (id, document_id, content, heading, embedding)
   VALUES ($1, $2, $3, $4, $5)`,
  [chunk.id, chunk.documentId, chunk.text, chunk.heading, toVector(vector)]
);

// Query: nearest neighbours by cosine distance.
const { rows } = await db.query(
  `SELECT id, content, heading, 1 - (embedding <=> $1) AS score
   FROM chunks
   ORDER BY embedding <=> $1
   LIMIT 20`,
  [toVector(questionVector)]
);

Note the LIMIT 20 rather than 5. Retrieve generously, then narrow. The next step explains why.

Step 4: reranking

This is the step most tutorials skip and the one that most reliably improves answers. Vector similarity is fast and approximate. It gets the right chunk into the top twenty far more reliably than it gets it into the top three, and the top three is what you have room to send.

A reranker is a slower, more accurate model that scores each candidate against the question directly rather than comparing pre-computed vectors. You run it over the twenty candidates and keep the best four.

const candidates = await vectorSearch(question, { limit: 20 });
const reranked = await rerank(question, candidates);
const context = reranked.slice(0, 4);

Two extra lines, one extra API call, and in my experience a larger quality jump than any amount of prompt rewriting. If you have a RAG system that "mostly works but misses obvious things", this is the first thing I would add.

Step 5: building the prompt

Now assemble. The thing to be deliberate about is what you tell the model to do when the retrieved context does not contain the answer.

const context = chunks
  .map((c, i) => `[${i + 1}] ${c.heading ?? ''}\n${c.content}`)
  .join('\n\n');

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 4096,
  system: [
    {
      type: 'text',
      text: [
        'Answer using only the numbered sources provided.',
        'Cite the source number in square brackets after each claim.',
        'If the sources do not contain the answer, say so. Do not use',
        'general knowledge to fill the gap.',
      ].join(' '),
      cache_control: { type: 'ephemeral' },
    },
  ],
  messages: [
    { role: 'user', content: `Sources:\n\n${context}\n\nQuestion: ${question}` },
  ],
});

The instruction to refuse rather than improvise is doing real work. A model given four irrelevant chunks and a question will usually produce a plausible answer from its training data, and that answer will look exactly like the grounded ones. Explicitly permitting "I do not know" converts a silent wrong answer into a visible retrieval failure, which is the one you can actually fix.

The cache_control on the system block is worth having from day one. The instructions are identical on every request, so caching them cuts input cost on the stable prefix. The details of that, and the ways it silently stops working, are in LLM API integration.

Where RAG systems actually fail

When answers are wrong, the instinct is to rewrite the prompt. It is almost never the prompt. Work backwards through the pipeline instead:

The cheapest debugging tool is logging the retrieved chunks alongside every answer. When something is wrong, you can see in one glance whether the model was given the right material and mishandled it, or was never given it at all. Nearly always, it is the second.

Hybrid search for the exact-match problem

Vector search is weak precisely where keyword search is strong. Ask about error code ERR_4021 and embeddings will helpfully return chunks about errors in general. Running both searches and combining the rankings fixes it, and Postgres can do both without a second system:

// Reciprocal rank fusion: combine two ranked lists without needing
// the two scoring scales to be comparable.
function fuse(vectorHits: string[], keywordHits: string[], k = 60) {
  const scores = new Map<string, number>();
  for (const list of [vectorHits, keywordHits]) {
    list.forEach((id, rank) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1));
    });
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

Rank fusion sidesteps the awkward problem that a cosine similarity of 0.82 and a BM25 score of 14.3 are not on the same scale and cannot be sensibly averaged. It only uses positions, which is why it works without tuning.

RAG evaluation, or you are guessing

This is the part I would push hardest on, and the part almost everyone skips. Without a test set you cannot tell whether a change helped, so every adjustment is a vibe.

The minimum viable version is thirty question and answer pairs written by hand, with the chunk that should have been retrieved recorded for each. That gives you the one metric that matters most: how often the correct chunk appears in the retrieved set at all. If it is missing, no improvement to the generation step can recover it. I have seen a system move from 60% to 90% on that number purely by changing the chunking strategy, with no other change.

Thirty examples in a JSON file, run in CI, will tell you more than any evaluation framework you adopt before you have the examples.

What I would build, in order

Structure-aware chunking, embeddings, pgvector, and a plain prompt. Then the thirty question test set, because everything after this point is unmeasurable without it. Then reranking, which is the biggest single win for the least work. Then hybrid search if your domain has identifiers, product codes or names in it.

Resist building a query-rewriting, multi-hop, self-correcting agent before the basic pipeline scores well. A retrieval system that finds the right chunk 90% of the time and a simple prompt beats an elaborate architecture on top of retrieval that finds it 60% of the time. If the model does need to take actions rather than just answer, that is a different design, covered in AI agents and tool calling.