Choosing a RAG architecture for enterprise knowledge

Almost every retrieval project we are called into has the same shape when we arrive: a working demo, an enthusiastic sponsor, and no way to tell whether last week’s change made it better or worse. The model is fine. The embedding is fine. What is missing is the part that makes it an engineering artefact rather than a demo — a way to measure it.

This is the architecture we start from, why each piece is there, and the decisions we would argue about on your particular corpus.

Start with the corpus, not the model

The single most common mistake is choosing an architecture before measuring the corpus. Retrieval quality is dominated by the interaction between document structure and query type, and both are properties of your business, not of the vendor you pick.

Before writing any pipeline code, we measure four things:

This takes about a day and it routinely eliminates half the design space. On one engagement it told us the correct answer was not RAG at all: the queries were 90% exact-identifier lookups against a structured system, and a Postgres index answered them at a fraction of the cost.

Chunking: structure first, tokens second

Fixed-size chunking with overlap is the default everywhere and it is nearly always the wrong first choice for enterprise documents, because it cuts through the semantic boundaries that make a passage answerable. Our order of preference:

1. Don’t chunk. If p95 document length fits in the embedding window, embed whole documents. Simplest thing that works.
2. Chunk on structure. Split on heading boundaries, clause numbers, or list items. Carry the heading path into the chunk text so the embedding knows where it came from.
3. Parent–child. Embed small chunks for retrieval precision; return the enclosing section for generation context. This is the single best quality-per-unit-complexity trick in the stack.
4. Fixed-size with overlap. The fallback for unstructured text, not the starting point.

Structural chunking with the parent–child pattern, in outline:

type Chunk = {
  id: string;
  documentId: string;
  parentId: string | null;   // section this chunk belongs to
  breadcrumb: string[];      // ["Master Agreement", "8. Liability", "8.2 Cap"]
  text: string;              // what we embed
  charStart: number;         // offsets back into the source document,
  charEnd: number;           // so a citation can be resolved to a span
};

function chunkBySection(doc: ParsedDocument): Chunk[] {
  const out: Chunk[] = [];

  for (const section of doc.sections) {
    // The section is the "parent" — retrieved for context, not embedded.
    const parentId = createId();
    out.push({
      id: parentId,
      documentId: doc.id,
      parentId: null,
      breadcrumb: section.breadcrumb,
      text: section.text,
      charStart: section.charStart,
      charEnd: section.charEnd,
    });

    // Children are embedded. Prefix the breadcrumb so an isolated clause still
    // carries its context into the vector.
    const prefix = section.breadcrumb.join(" > ");
    for (const unit of splitIntoUnits(section, { maxTokens: 320, minTokens: 64 })) {
      out.push({
        id: createId(),
        documentId: doc.id,
        parentId,
        breadcrumb: section.breadcrumb,
        text: prefix + "\n\n" + unit.text,
        charStart: unit.charStart,
        charEnd: unit.charEnd,
      });
    }
  }

  return out;
}

Keeping charStart and charEnd is not optional. It is what later lets a citation point at a highlightable span in the original document rather than at a chunk ID no reviewer can verify.

Storage: one Postgres, two indexes

Unless you are past roughly 50 million chunks, you do not need a dedicated vector database. Postgres with pgvector and a full-text index gives you both halves of hybrid retrieval, in one transaction, with the access control you already have. The operational saving is significant and the quality cost is zero.

create extension if not exists vector;
create extension if not exists pg_trgm;

create table chunk (
  id            uuid primary key,
  document_id   uuid not null references document(id) on delete cascade,
  parent_id     uuid references chunk(id) on delete cascade,
  tenant_id     uuid not null,
  breadcrumb    text[] not null default '{}',
  text          text not null,
  char_start    int  not null,
  char_end      int  not null,
  embedding     vector(1536),
  -- Generated, so it can never drift out of sync with text.
  tsv           tsvector generated always as (to_tsvector('english', text)) stored,
  created_at    timestamptz not null default now()
);

-- Dense. Tune lists ~= sqrt(rows); probes trades recall for latency at query time.
create index chunk_embedding_idx on chunk
  using ivfflat (embedding vector_cosine_ops) with (lists = 2000);

-- Lexical.
create index chunk_tsv_idx on chunk using gin (tsv);

-- Every retrieval is tenant-scoped. Enforced in the database, not the application.
alter table chunk enable row level security;

create policy chunk_tenant_isolation on chunk
  using (tenant_id = current_setting('app.tenant_id')::uuid);

Two notes that cost people weeks. First, the tsvector is a generated column — if you populate it in application code it will eventually disagree with text, and the failure is silent. Second, put row level security on the chunk table from day one. Retrofitting tenant isolation into a retrieval path after launch means re-auditing every query, and a cross-tenant leak in a knowledge system is the kind of incident that ends the project.

On index choice: ivfflat is cheaper to build and adequate to roughly ten million rows. hnsw gives better recall at a given latency but costs considerably more memory and build time. Start with ivfflat, measure, and move only if recall is the bottleneck.

Hybrid retrieval, fused with RRF

Dense-only retrieval is the default in tutorials and it underperforms on every enterprise corpus we have measured — typically by 10 to 20 points of recall@20. The reason is mundane: business queries are full of identifiers, part numbers, proper nouns, and internal jargon, which are exactly what embeddings smooth away and exactly what BM25 nails.

Run both, then fuse. Reciprocal Rank Fusion is the right default because it needs no score normalisation between two systems whose scores are not comparable, and no tuning beyond a single constant.

-- Two independent rankings, fused by reciprocal rank. K=60 is the
-- conventional constant and we have never needed to move it.
with dense as (
  select id, row_number() over (order by embedding <=> $1) as rank
  from chunk
  where parent_id is not null
  order by embedding <=> $1
  limit 100
),
lexical as (
  select id, row_number() over (
           order by ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) desc
         ) as rank
  from chunk
  where parent_id is not null
    and tsv @@ websearch_to_tsquery('english', $2)
  limit 100
)
select
  coalesce(d.id, l.id) as id,
  coalesce(1.0 / (60 + d.rank), 0) +
  coalesce(1.0 / (60 + l.rank), 0) as rrf_score
from dense d
full outer join lexical l on d.id = l.id
order by rrf_score desc
limit 40;

Note parent_id is not null: we retrieve against children and expand to parents afterwards. And note that both halves run in one query against one database — no fan-out, no two-system consistency problem.

A weighted variant is available if one channel is clearly stronger on your corpus, but do not reach for weights before you have the evaluation harness to justify them. Untuned weights chosen by intuition perform worse than unweighted RRF in our experience.

Reranking: the best addition, and the easiest to over-buy

A cross-encoder reranker over the top 40 fused candidates is usually the largest single quality jump available after hybrid retrieval — in our measurements, 8 to 15 points of nDCG@10. It also adds 150–600 ms and a per-query cost, so it belongs in a latency budget, not in a feature list.

The pattern that keeps it affordable:

• Retrieve 100 per channel, fuse to 40, rerank 40, pass the top 6–8 parents to generation. Reranking more than ~50 candidates has diminishing returns on every corpus we have tested.
• Cache reranker output keyed on (normalised_query, candidate_id_set). Enterprise query distributions are far more repetitive than public ones — we routinely see 30–40% cache hit rates.
• Skip the reranker when the fused score distribution has a clear winner. If the top result’s RRF score exceeds the runner-up by a wide margin, the ordering is not in doubt.

Grounding: enforce citations, do not request them

“Cite your sources” in a system prompt is a preference, not a control. If an unsourced answer is unacceptable in your domain — and in contracts, clinical, or financial contexts it is — the pipeline has to fail closed.

We make the model emit claims and citations as structured output, then verify every citation against the retrieved set before the answer is allowed out:

type Claim = {
  text: string;
  citations: { chunkId: string; quote: string }[];
};

function verify(claims: Claim[], retrieved: Map<string, Chunk>): Verdict {
  for (const claim of claims) {
    if (claim.citations.length === 0) {
      return { ok: false, reason: "uncited_claim", claim: claim.text };
    }

    for (const c of claim.citations) {
      const chunk = retrieved.get(c.chunkId);

      // The model cited something we never retrieved.
      if (!chunk) {
        return { ok: false, reason: "hallucinated_source", chunkId: c.chunkId };
      }

      // The quote must actually appear in the source, not merely resemble it.
      if (!normalise(chunk.text).includes(normalise(c.quote))) {
        return { ok: false, reason: "quote_not_in_source", chunkId: c.chunkId };
      }
    }
  }
  return { ok: true };
}

A failed verdict does not get repaired silently. It routes to the review queue with the reason attached, which means your reviewers see the model’s actual failure modes instead of a laundered version of them. That data is how you improve the retrieval layer.

The evaluation harness comes first

This is the section that separates systems from demos, and it is the one most often skipped. Build it before the pipeline. Without it you cannot answer the only question that matters — did that change help? — and every subsequent decision is taste.

The golden set

150 to 300 query–answer pairs, written by people who do the job, sampled from real query logs where they exist. Each pair records the query, the passages that should be retrieved, and an acceptable answer. Building it costs about a week of a domain expert’s time and it is the highest-return week in the project.

Stratify it deliberately, or your average will hide your failures:

• Lookup queries with exact identifiers
• Paraphrased queries with no lexical overlap
• Multi-hop queries needing two or more documents
• Unanswerable queries.Usually 15–20% of the set. A system that never says “not in the corpus” is worse than useless in a regulated setting.

What to measure

Wire it into CI

The harness is only useful if it blocks merges. A prompt edit, a chunk-size change, and a model upgrade are all the same category of change, and all three should have to clear the same gate.

# Fails the build on regression against the committed baseline.
# Prompts and chunking parameters are code, and are gated like code.
name: rag-eval
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run eval -- --set golden --out results.json
      - run: |
          npm run eval:compare -- \
            --baseline eval/baseline.json \
            --candidate results.json \
            --min-recall-at-20 0.92 \
            --min-ndcg-at-10 0.78 \
            --min-citation-precision 0.98 \
            --min-abstention 0.90 \
            --max-p95-ms 2500 \
            --max-regression 0.01

The --max-regressionflag matters as much as the floors. It catches the change that lifts the average while quietly destroying one stratum — which is what most “improvements” actually do.

Latency and cost budget

Indicative figures for a corpus of roughly two million chunks on a single Postgres instance, useful for sizing an argument rather than a quote:

What would make us change our minds

A decision record is only honest if it says what would overturn it. On this architecture:

• Drop hybrid for dense-only if lexical retrieval contributes under 3 points of recall@20 on the golden set. On a corpus of fluent prose with no identifiers, it sometimes does.
• Leave Postgres past roughly 50 million chunks, or when index rebuild time starts to constrain the ingest schedule.
• Drop the reranker if nDCG@10 gains fall below 5 points, or if the latency budget tightens below about 1.5 s end to end.
• Abandon RAG entirely if the query mix turns out to be predominantly exact-identifier lookup against structured data. This is more common than the literature suggests, and an index is cheaper than a pipeline.

The order we build it in

1. Corpus measurement — one day, eliminates half the design space
2. Golden set — one week of a domain expert, highest-return week available
3. Evaluation harness and CI gate, against a deliberately naive baseline
4. Storage schema with row level security from the first migration
5. Structural chunking, measured against the baseline
6. Hybrid retrieval and RRF fusion
7. Citation verification, failing closed into a review queue
8. Reranking, last, with the latency budget already agreed

The ordering is the point. Every team that builds the pipeline first and the harness later ends up rebuilding the pipeline, because they cannot tell which parts of it were ever load-bearing.