Context, Memory & Why AI Forgets

Session 1 ended with a planted question: “If every new chat starts from zero, how do companies build AI systems that actually remember things?” That question is this session's entire agenda - and by the end, you'll have the complete answer.

Here is what Session 1 established: models are stateless, weights are frozen at training, and the context window is the only “memory” a model has for any given request. When you close the tab, everything disappears. The model doesn't carry forward a single word.

The pre-session assignment asked you to map a repetitive task to an AI opportunity. That task is your working example for this entire session - the documents it would need, the context it would require, the memory it would have to persist. Keep it in mind.

Statelessness is not a bug in LLMs. It is the design. Every API call is completely independent. No knowledge, no context, no history flows between sessions unless you explicitly engineer it to do so.

Recall the frozen weights point from Session 1: at inference time, the model's parameters don't change. Nothing about your conversation affects the underlying model. The weights that existed when you started are identical when you finish.

There's a critical distinction most people miss: “the model doesn't remember” is a model property. “The system doesn't persist” is an engineering choice. You cannot change the first. You absolutely can change the second - and that's what the rest of this session is about.

When you use a product that “remembers” your name, your preferences, or your past questions - that product is injecting stored context into the current prompt. The model isn't recalling anything. A database retrieved your profile, and it was pasted into the context window before your message was processed. What looks like memory is text injection.

Session 1 introduced the context window. This session goes deeper - because understanding how it fills, what gets dropped, and why position matters is the foundation of every memory engineering decision you'll make.

Think of the context window as working memory for one request. Everything the model knows, for this specific call, must live inside it. What doesn't fit simply does not exist as far as the model is concerned. No warning. No error. Silent truncation.

There are exactly three approaches to making an AI system retain information across a session, across sessions, or across all users. They differ in cost, complexity, freshness, and what kind of knowledge they handle well. Understanding all three - and when each is right - is the core design skill this session builds.

Session 1 introduced the knowledge cutoff. Now that you understand the three memory approaches, you can see clearly why it matters and how to solve it.

The cutoff means the model genuinely does not know about anything that happened after its training data was collected. But the more practical problem for most enterprises isn't news events - it's proprietary data. Your internal documents, last month's pricing, updated policies, customer records: the model has never seen any of it.

The naive solution - paste the document into the prompt - works for one document, one question. It fails immediately when you have hundreds of documents, when the right document isn't obvious, or when the document is longer than available context. RAG is the scalable solution.

RAG - Retrieval Augmented Generation - is the standard enterprise architecture for giving AI access to proprietary or current information without retraining. Here is the complete pipeline, step by step.

Session 1 introduced the idea that tokens become vectors - that “King − Man + Woman ≈ Queen” is a real result in vector space. Embeddings are the foundation of RAG retrieval. Here is the complete picture.

An embedding is a list of numbers - typically 768 to 1,536 floating-point values - that encode the meaning of a piece of text. The embedding model is trained on massive amounts of text to learn that texts with similar meaning should produce similar numerical vectors. The numbers themselves have no direct interpretation - only their relationships to each other matter.

Similarity is measured using cosine similarity: how similar are the directions of two vectors in this high-dimensional space? Two identical texts have cosine similarity of 1.0. Unrelated texts have cosine similarity near 0. Semantically related but differently-worded texts score somewhere in between - which is what makes semantic search work.

Why does this beat keyword search? Because it captures meaning, not just word overlap.

A distance visualisation. Imagine a simplified 2D version of embedding space. A query about “vacation leave policy” produces a vector. In the document store:

A vector database stores embedding vectors alongside their source text, and enables fast similarity search across millions of entries. When your RAG system asks “what documents are most similar to this query?”, the vector database answers that question - typically in 10–100 milliseconds.

What makes a vector database different from a regular database: regular databases are optimised for exact lookups (“find the row where ID = 4821”). Vector databases are optimised for approximate nearest-neighbour search (“find the 10 vectors most similar to this query vector”). The underlying algorithms - HNSW, IVF, PQ - are specialised for this problem.

Not all RAG implementations are equal. There are four patterns, each with different complexity, cost, and retrieval quality. Start simple and graduate as your requirements demand it.

Session 1 introduced conversation memory briefly. Now that you understand the full context picture, here are the four production patterns - and when each one is appropriate.

Every token in your context window costs money and competes for the model's attention. Raw documents injected without pre-processing are almost always the wrong approach. A 20-page PDF cleaned down to 4 relevant paragraphs will outperform the raw PDF every time.

Context hygiene is not optimisation - it is correctness. A model given clean, relevant context consistently outperforms the same model drowning in noise. Preprocessing is the highest-leverage, lowest-cost improvement available in any RAG system.

What to strip before ingesting: headers and footers (every page says “CONFIDENTIAL - Page 12 of 47”), repeated boilerplate, table of contents (it duplicates structure without adding content), excessive whitespace, page numbers.

What to keep: the substantive content, section titles (important for context), document title and date as metadata, author or source attribution.

These are not alternatives in a hierarchy - they are tools for different problems. The decision of which to use depends on four factors: what kind of knowledge is needed, how frequently it changes, what volume of calls you're running, and what your control requirements are.

Each exercise takes 5–10 minutes and demonstrates something you cannot fully grasp from reading alone. No code required for any of them.

Enterprise AI products typically need all three memory tiers simultaneously. Understanding which tier handles which need prevents the common mistake of trying to solve everything with a single RAG index.

Memory has a token cost. Every technique discussed in this session - RAG, conversation history, user profiles - adds tokens to every API call. At low volume this is trivial. At enterprise scale it becomes the dominant line item. Know the numbers before you design.

These are the five patterns that consistently produce bad RAG output. Each one has a specific, practical fix.

RAG is not a reliability fix. It is an information injection mechanism. Understanding this distinction prevents a dangerous overconfidence that frequently leads to production failures.

RAG injects context into the prompt. It does not make the model more accurate within that context. The model is still predicting the next token - it is still capable of misreading, misquoting, or partially interpreting the retrieved chunks. The retrieval can be perfect and the generation can still be wrong.

Work through these questions yourself, or bring them to the group session. They are designed to connect the mechanics you've just learned to the real use cases you identified before this session.

The goal is not a perfect design - it is to force the architectural questions. Most teams never ask “how would I chunk this?” until they are already in production and retrieval is failing. Asking it now, with a real use case, makes Session 3 immediately applicable.