Memory brings stateful continuity to agentic systems. Unlike an infinitely long context window, it evolves and is portable across sessions. It could be short-term or long-term, storing user preferences and abstracted knowledge.
In the Advanced LLM Agent open course by Berkeley, a neurobiology-inspired long-term memory is introduced. It includes two main components: memory encoding and memory retrieval.
Offline Indexing (Memory Encoding):
An instruction-tuned LLM processes a corpus of text and extracts a schemaless knowledge graph (KG) from each passage using Open Information Extraction, treating noun phrases as nodes and their relationships as edges. This KG acts as an artificial hippocampal memory index. Retrieval encoders then enrich the KG by adding edges between semantically similar noun phrases, simulating pattern completion. The result is a unified graph that integrates knowledge across the entire corpus, structurally mirroring the brain's memory formation process.
Online Retrieval (Memory Retrieval):
When a query is given, an LLM identifies key named entities and matches them to related nodes in the KG. These nodes serve as cues to initiate retrieval. HippoRAG then runs Personalized PageRank (PPR) from these nodes to explore the KG, distributing relevance across connected nodes and mimicking context-based memory recall. Node specificity adjusts the influence of each seed node, favoring more distinctive concepts. The final PPR distribution determines which passages are most relevant based on the density of high-ranking KG nodes they contain.
These systems offer significant advantages: continuous learning without catastrophic forgetting, efficient handling of partial queries, and transparent retrieval processes. With these, it enables multi-hop reasoning in a single retrieval system.
However, they also present challenges: implementation complexity, resource requirements, and potential scalability issues with extremely large datasets.
Reference: https://mem0.ai/blog/memory-in-agents-what-why-and-how/
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models