How Do You Implement Access Control in a RAG Pipeline?

Attach provenance metadata to every chunk at ingestion, which workspace owns it, where it came from, who created it and who its audience is, propagate that metadata through every derivation step, and resolve the requester's authority into a filter that runs as part of the vector query itself. Never as a post-processing step over the results.

That is the whole answer, and each clause of it is a place where real systems fail. Chunks lose their scope during summarisation. Filters get applied after the search instead of inside it. Data that predates the policy gets treated as public because the alternative made search results look thin.

I'm Mahmoud Zalt, an AI architect. I founded Sista AI, and retrieval permissions are the single most common thing I find broken when reviewing an AI system that already works well enough to have customers.

This is part two of a series. Part one covered why prompts are not permissions and the four checks every sensitive operation should pass.

A File Has an Owner. A Memory Has Nothing.

Start with the asymmetry that makes this hard.

A file arrives with structure already attached. It has an owner, a location, a sharing dialog, an audit trail and decades of prior art in how to secure it. Every engineer on your team already knows how to reason about it.

A fact your AI learned in a conversation has none of that.

It is a sentence. Maybe a vector. It was derived at 11pm from something a founder typed, and by the time it reaches your retrieval layer it may be nothing more than a string, a float array and a tenant id.

Tenant id is not an access policy. It is the beginning of one, and most AI products stop there, which is how a company ends up with a system where anyone who can log in can retrieve anything anyone in the company ever said to the assistant.

If RAG is new to you, the short version is that the system searches your own content and pastes the best matches into the model's prompt. Which means retrieval is not a search feature. Retrieval is the thing that decides what enters the context window, and therefore what the model can possibly say.

Every Chunk Needs a Passport

The fix is not exotic. It is discipline about identity. Every sensitive source should carry a durable provenance record that travels with it, permanently, through every transformation:

source_id
workspace_id
origin          conversation | upload | integration | tool_output | generated
created_by      principal that produced it
audience        organisation | named_principals[] | private
derived_from[]  parent source ids
policy_version
indexed_at

That record is what retrieval filters on. Not the text, not the vector. The passport.

And one rule holds the whole thing together:

Note the derived_from field. Lineage is not decoration. It is the only way to answer the question you will eventually be asked under time pressure: this summary contains something it should not, where did that sentence come from, and what else did the same source touch?

Derivation Is Where Permissions Quietly Die

This is the part that bites in production, and it does not announce itself.

Data in an AI system does not sit still. It gets chunked, summarised, embedded, re-ranked, cached, rolled into a weekly digest, distilled into a memory episode, folded into a user profile, used as a few-shot example. Every one of those steps is an opportunity for a private fact to shed its label and re-enter the system as an ordinary organisation-wide string.

The aggregation case is the sharpest edge. Summarise forty items, thirty-nine of them public and one of them restricted, and you have created a brand new artifact that no policy was ever written for. If that summary does not inherit the restriction of the one restricted input, you have laundered a private fact into a company-wide document, with the citation helpfully removed so nobody can trace it back.

So the invariant has to be enforced at the point of derivation, not patched afterwards. A practical test: if a pipeline stage cannot state where its output came from, that stage is a leak waiting for a witness.

Why Filtering After Retrieval Is Not Access Control

There is a shortcut that looks correct in a demo and fails in production: retrieve the top matches, then drop the ones the requester is not allowed to see.

Two problems, and the second is worse than the first.

The obvious one: recall collapses silently. You ask for ten results, get ten, discard seven, and answer confidently from three. Nobody sees the degradation, because the model's tone does not change when its evidence gets thin. This is one of the quieter reasons RAG systems return wrong answers, and it is invisible in evaluation runs done with an admin account.

The real one: the data was already read. Post-filtering means the unauthorised rows were fetched, ranked, and held in process memory. They are now one logging statement, one debug trace, one error path or one cache layer away from the requester. You did not prevent access. You prevented display. Those are not the same thing, and only one of them survives a security review.

Scope belongs in the query predicate. The requester's authority is resolved before the search runs, and the search cannot return what the requester cannot reach. If your vector store cannot express that, it is a constraint on your architecture, not a licence to filter late.

What to Do With Data of Unknown Provenance

Here is the decision that separates serious systems from optimistic ones.

When a piece of data has no trustworthy scope, an old row, an unlabelled import, a chunk from a pipeline that predates the policy, you have exactly two available defaults.

Treat it as organisation-wide, because it is probably fine and the alternative breaks search.

Or treat it as inaccessible until proven otherwise.

The first default is how cross-tenant incidents happen. The second costs you a migration and some uncomfortable weeks where results look thinner than they should.

Take the second every time. Fail closed, then do the provenance work to re-open what deserves to be open, deliberately, with a record of the decision.

I will be honest that this is real engineering rather than a config flag. Legacy data, older integrations and anything indexed before the policy existed all need deliberate backfill. It is not glamorous work and it does not demo. It is also the difference between a system whose security you can describe and one whose security depends on data nobody can vouch for.

Revocation Is a Write Path, Not a Checkbox

The last piece people underestimate.

When access is revoked, the question is not whether the sharing dialog updated. The question is what else in your system is still holding that authority.

Retrieval indexes. Precomputed digests. Cached agent state. Warmed context. Background jobs already in flight. Export queues. A summary generated last week from data the requester could see last week and cannot see now.

Revocation has to reach all of it, and it has to reach it immediately, because the gap between revoked in the interface and revoked in retrieval is exactly the window an incident lives in.

The mental model worth adopting: a permission change is not a state update on one row. It is an invalidation event that fans out across every derived surface.

A Retrieval Permissions Checklist

What I look for when reviewing a retrieval layer that serves more than one person:

That last row catches more problems than the rest combined. Teams evaluate retrieval as an admin, ship, and then discover that for an ordinary member the system answers half the questions from a third of the evidence. Whether you even need this layer depends on your setup, which I covered in do you need RAG for your AI agent, and the broader integration question in RAG inside an AI agent.

Frequently Asked Questions

Is metadata filtering in a vector database enough for multi-tenant security?
Only if the filter runs inside the search, is derived from a server-resolved identity rather than a client-supplied value, and every chunk actually carries correct scope metadata. The usual failure is not the filter itself, it is chunks that lost their scope somewhere upstream.

What happens when an AI summarises documents with different permissions?
The summary must inherit the most restrictive scope among its inputs. Otherwise aggregation becomes a laundering path that turns restricted facts into organisation-wide artifacts nobody wrote a policy for.

Should each tenant get a separate index or collection?
Separate indexes give you a strong isolation boundary and are worth it at the tenant level. They do not solve the harder problem, which is access control between people inside the same tenant. You still need per-chunk scope.

How fast does revoked access need to take effect in a RAG system?
Immediately, and across every derived surface: indexes, caches, precomputed digests, warmed context and in-flight jobs. Treat revocation as an invalidation event rather than a row update.

Do embeddings leak the content they were generated from?
Treat them as if they do. Embedding inversion research keeps improving, and in any case the vector is stored next to enough metadata to be useful to someone who should not have it. Scope embeddings exactly as you scope the source text.

Three Things to Take Away

One. Give every chunk a passport at ingestion and make it survive every derivation. Provenance you add later is provenance you are guessing at.

Two. Filter inside the query. Post-filtering is display control wearing the costume of access control.

Three. Fail closed on unknown data, then earn back access deliberately. The version of this decision you make under deadline pressure is always the wrong one.

Part three of this series covers delegated authority and audit logs: what the agent is allowed to do once it has legitimately retrieved something, and how to prove afterwards that enforcement was real. If your retrieval layer needs a review before it serves a second customer, that is a common starting point for an agent engagement.

Get your retrieval layer reviewed ->