AI Agents · Memory architectures

Why this matters

LLMs are stateless. Every call starts from zero. Production agents need to remember — across turns, across sessions, across users. There are five distinct memory types; mixing them up is a common cause of confused-feeling assistants.

Five memory types

Short-term: keeping the conversation window alive

• Sliding window: keep last N messages, drop the rest. Cheap, simple, drops important context randomly.
• Sliding window + summary: keep last N messages PLUS a rolling summary of older ones. Best default for chat apps.
• Relevance retrieval: embed all messages, retrieve the K most relevant to the current turn. Better for very long sessions; costs latency.
• Hierarchical compression: re-summarise older summaries. Tree of compressed memory.

# Simple sliding window + summary pattern
def get_context(session_id, current_query):
    msgs = db.messages.find(session_id).order_by(time).limit(20)
    if len(msgs) >= 20:
        older = db.messages.find(session_id).older_than(msgs[0].time)
        summary = summarise_with_llm(older)   # cached; only recompute when stale
        return [{"role":"system","content":summary}, *msgs]
    return msgs

Long-term: vector-store-backed semantic memory

Distil important facts from conversations into discrete "memories", embed them, retrieve relevant ones at the start of each new turn. This is what Mem0, Zep, and the "memory" features in ChatGPT do under the hood.

sequenceDiagram
    participant U as User
    participant A as Agent
    participant M as Memory store
    participant LLM
    U->>A: New message
    A->>M: retrieve(top_k=5, query=msg)
    M-->>A: 5 relevant memories
    A->>LLM: [memories + current_msg]
    LLM-->>A: response
    A->>LLM: extract_facts(conversation)
    LLM-->>A: new facts to store
    A->>M: upsert(new_facts)
    A->>U: response

Memory extraction is the hard part

Storing every message is noise. Storing nothing is amnesia. The trick is a dedicated memory extractor — a small LLM call after each turn that decides what's worth keeping:

EXTRACT_PROMPT = """
Given this conversation turn, extract durable facts about the user worth remembering.
Return JSON: { "facts": [{"text": "...", "category": "...", "confidence": 0-1}] }
If nothing is worth keeping, return { "facts": [] }.

Examples of facts worth keeping:
- "User is vegetarian"
- "User is hiring for a backend role"
- "User's company is Acme"

Examples NOT to keep:
- Greetings, pleasantries, small talk
- Information about other people, only about THIS user
- Anything the user might want to change or undo

Conversation:
{conversation}
"""

Privacy and the forget button

The token budget for memory

Every memory you stuff into the prompt costs tokens. A 4k context with 2k of memory leaves 2k for actual conversation. Production budgets:

• ~300 tokens of summary (sliding window summary)
• ~500 tokens of retrieved long-term memories (top 3-5)
• ~500 tokens of recent messages (last 4-6)
• Rest: current message + room for response