A model knows what it was trained on: a snapshot of public text, frozen at a point in time. It has never seen your data, your internal docs, or how your product actually works. Ask it about those and it will answer anyway, confidently and wrong. This chapter gets the model answering from your data instead of inventing it.
7.2.1The model doesn't know your world
The model runs in a sealed box. It cannot reach your database, read your files, or look anything up on its own, so anything specific to you is a blank it fills with a plausible guess.
The fix is not to retrain it on your data, which is slow and expensive. Instead, when you ask a question, you fetch the few pieces of your data that answer it, then hand those to the model. That pattern is RAG, retrieval-augmented generation: retrieve the relevant facts, then let the model generate its answer from them.
7.2.2Turn text into vectors you can search
First you split your data into chunks, small self-contained passages: a paragraph of a doc, one help article, a single record. Then you turn each chunk into an embedding, a long list of numbers that captures its meaning, produced by a model built for exactly this.
The trick is that meaning becomes distance. Chunks about the same idea get similar numbers and sit close together. So "reset my password" lands near "I forgot my login", even with no shared words.
7.2.3Store and search them in a vector database
A vector database stores every chunk's embedding and is built to answer one question fast: which stored chunks sit closest to this one? You load your embeddings in once, then query it as often as you like.
To search, you embed the question the same way, then ask for its nearest matches. It matches on meaning, not exact keywords, so a question worded nothing like your docs still finds the right passage.
7.2.4Retrieve, then answer
You already know how to write a prompt; retrieval just changes what goes into it. You paste the top matches in, above the question, with a line telling the model to answer only from them.
Now the model reads real facts instead of reaching for a guess. Keep the retrieved set small: the best three to five chunks, not everything you own. Too much context buries the answer and costs more.
Rule of thumb: if the answer is not in the chunks you retrieve, the model cannot know it. Retrieval quality is the whole game.
Start simple, and if basic retrieval is not accurate or cheap enough, reach for the known upgrades and hand them to your agent by name. Add plain keyword matching alongside the vector search (hybrid search), reorder the retrieved chunks so the best ones lead (reranking), and cache repeated answers to cut cost. For complex, connected data, fancier variants like GraphRAG and multi-hop retrieval exist, but only reach for them when the simple pipeline falls short.
This prompt sets the whole pipeline up on your data:
Do this now: paste the prompt, point it at one folder of your docs, and ask a question those docs answer that the model could never answer on its own.
