RAG Architecture Design Theory and Conceptual Organization in the Age of AI Agents: 7 Patterns

Greetings from the island nation of Japan.

This article attempts a rather ambitious feat: bringing a semblance of order to the glorious chaos that is Retrieval-Augmented Generation (RAG) Architecture in the age of AI Agents.

One might assume, looking at a Large Language Model, that it is simply a clever box that produces answers. A delightfully convenient illusion. The reality, as we engineers know, involves navigating a minefield of terminology and the structural integrity of something resembling a digital 'Spaghetti Junction' of data pipelines.

When the brief arrives to build an "AI Agent," one must resist the urge to simply nod politely and immediately book a one-way ticket to a remote island. (Alas, as I already reside on one, that option is closed.)

Instead, one must embark upon the meticulous, yet necessary, task of separating the 'Agentic Workflow' (the noble intention, or The What) from the 'Agentic Architecture' (the tiresome, costly engineering, or The How). Failure to do so, I assure you, is simply not cricket.

Having prepared myself a rather weak cup of tea—a metaphor, perhaps, for the often-diluted knowledge passed down in AI discussions—let us proceed to the seven essential patterns that will allow you to build something scalable, rather than merely something shouty.

I trust you will find this structural guidance to be, at the very least, adequate.

1. Clarifying Ambiguity and a Paradigm Shift in RAG Design

1.1. The Evolution of RAG and Confusion in Design Concepts: Why Terminology Needs Clarification

Retrieval-Augmented Generation (RAG), which enhances the capabilities of Large Language Models (LLMs) with external knowledge, has rapidly evolved as a foundational technology for AI applications. It's evolving so fast it's scary, and I'm struggling to keep up.

I need to organize my thoughts in this article, especially since I mentioned there were four approaches...

4.1. The Essence of Data Preprocessing: The Importance of Appropriate Chunking Strategies and Metadata Utilisation

Custom Chunking Strategies

Chunking strategies form the bedrock of RAG systems. Given the diverse nature of production environment data, it is essential to divide chunks so that each retains self-contained information as a logical unit, rather than mechanically cutting words or sentences midway⁹. Standard chunkers (e.g., Unstructured.io) provide a starting point, but building a custom chunking flow is required to accommodate domain-specific data structures and formats (particularly corporate data)⁹.

Corporate data often suffers from rather idiosyncratic storage methods (a veritable parade of wildly unconventional formats like bizarre Excel files, bizarre Word documents, and excessively fiddly PDFs). While type conversion is important, it would be beneficial to address these issues too.

Metadata Injection

While early approaches often pass only the chunked text to the LLM, experimental results demonstrate that combining relevant metadata (e.g., document title, author, section information) with the chunked text and injecting this as context into the LLM significantly improves response quality⁹. This helps the LLM gain a deeper understanding of the source and context of the provided information, enabling it to generate more reliable (grounded) responses. When I first learnt this, it really gave me an adrenaline rush.

4.2. Techniques for Dramatically Improving Search Accuracy

In large-scale systems, reliably presenting the information users seek at the top of results directly impacts the system's credibility. That said, I think it's common to encounter phenomena where this isn't the case during verification.

The Overwhelming ROI of Reranking

Reranking is often described as the ‘five lines of code with the highest value’ among strategies to add to production RAG systems, offering remarkably significant benefits relative to its ease of implementation⁹. Adopting a reranker can compensate for weaknesses such as suboptimal initial retriever configuration or insufficient vector embedding quality. This is achieved by inputting a sufficient number of chunks (e.g., 50 chunks) initially⁹. This demonstrates the practical lesson that improving search quality should be prioritised before undertaking complex architectural changes.

Practical Implementation of Hybrid Search

Implementing Hybrid Search is a crucial step towards broadening search coverage. By combining semantic search with keyword search, it achieves both semantic accuracy and word-level precision¹². In a case study involving 5 million documents, selecting a vector database (e.g., Turbopuffer) that natively supports keyword search contributed to efficient Hybrid Search implementation in large-scale environments⁹. Reciprocal Rank Fusion (RRF), as mentioned earlier, is typically used for result integration¹².

This was genuinely helpful as my own thinking was starting to become rather rigid; I felt I'd gained some valuable insights.

4.3. Query Processing to Unlock LLM Capabilities: Advanced Query Generation and Routing

Advanced RAG systems do not merely accept queries; they optimise the queries themselves and manage the system's limitations.

Query Generation

The last query entered by the user may not capture the full context. To compensate, an effective approach involves using the LLM to review the entire conversation thread and generate multiple semantic queries or keyword queries in parallel ⁹. Executing these multiple generated queries concurrently and passing the results to the relancer ensures broader search coverage, including potential contextual elements. This is something I've experienced quite a lot in practice. I feel that in real-world settings and with users, there are far more short, directive phrases like ‘Do ◎◎’ or ‘△△!’ than one might expect, making it difficult to grasp the context... I think it's quite fundamental that how well instructions are given in the first place significantly impacts how effectively AI is utilised. This strategy of technically compensating for the ambiguity in user instructions is, I believe, where the true value of LLM-based query generation lies.

Query Routing

Defensive design is indispensable for ensuring system robustness. This is common knowledge and practically a given by now! Query routing is the mechanism whereby a RAG system detects queries outside the knowledge base's scope (e.g., tasks like ‘summarise this document’ or ‘who wrote this article’, which fall under processing or metadata extraction rather than information retrieval) and, instead of executing the full RAG pipeline, performs a separate, simpler API call or transfers the query to an LLM⁹. This avoids unnecessary RAG execution, optimising both cost and latency. Whilst a complex element of the agentic architecture, it is a fundamental strategy essential for stable, large-scale production deployment. There are various approaches to defence design.

ROI Analysis of Production RAG Optimisation Strategy (Based on a 5 Million Document Case Study)

5. Practical Design Guide: Combining RAG Architectures and Conclusions

5.1. Design Approach for Complex Requirements: Combining RAG Architectures

In real-world system development, RAG design is not confined to a single architecture pattern but is realised as a modularised system combining multiple strategies¹. Frankly, I suspect survival would be tough otherwise. When comparing from a product quality perspective, the superior approach is clearly preferable.

Successful case studies in large-scale systems demonstrate that a multi-layered approach is key: placing high-precision search techniques like Query Generation or Hybrid Search at the front end of the workflow, refining results via a Reranker, and then routing them to specific RAG modules via an Agentic Router⁹.

Within this design philosophy, Agentic RAG assumes the role of the orchestration layer for the entire RAG pipeline. For example, the Agentic Router can dynamically determine which RAG module to invoke—Hybrid RAG, Multimodal RAG, or Graph RAG—based on the user's query content. The Agentic architecture sits atop specialised RAG modules, functioning to enhance the adaptability and flexibility of the entire system.

5.2. Decision Matrix: Criteria for Architecture Selection

When selecting a RAG architecture, I believe evaluation can be conducted based on the following four primary design axes:

1. Data Properties: Whether the data being handled is text-only, multimodal data including images or audio, or contains complex relationships between entities. This determines the necessity of implementing Multimodal RAG or Graph RAG.




2. Required Task Autonomy: Whether queries can be resolved through simple question-answering, or whether step-by-step planning like ReAct or autonomous use of external tools is required. This determines the level of Agentic RAG needed (Router-based or Multi-Agent-based).




3. Performance and Cost: The response time, throughput, and computational resources required of the system. The level of high-ROI Reranking or Hybrid Search should be considered first.




4. Explainability and Trustworthiness: Is the ability to trace the reasoning behind generated answers and verify their reliability required? For use cases involving complex reasoning, adopting Graph RAG offers advantages¹⁹.

RAG has significantly increased the amount of thought required, even for a single word, while simultaneously expanding the available options. This area feels like a real showcase for technical prowess and a potential competitive edge, though it remains somewhat opaque.

5.3. Summary and Future Directions

Designing a RAG system is not merely an integration of technical components, but a decision-making process grounded in conceptual clarity and strategic optimisation. Designers must first rigorously distinguish between the “agentic workflow (what)” and the “agentic architecture (how)”, understanding whether the locus of control resides in fixed code paths or within the LLM's dynamic decision-making capabilities.

In practical terms, it is crucial to prioritise high-ROI search quality enhancement strategies—such as Reranking, Query Generation, and Hybrid Search—before implementing complex agentic architectures, thereby establishing a solid foundation for retrieval quality. This is because many challenges in RAG implementation projects stem not from a lack of advanced architecture, but from insufficient basic search accuracy. Ultimately, it boils down to the fact that feeding it rubbish isn't going to work, is it?

The future evolution of RAG is predicted to converge towards more flexible and adaptable Agentic Modular RAG systems, where diverse specialised modules are orchestrated by advanced autonomous agents. Or rather, I suspect the AGI trend is now unstoppable. ChatGPT Atlas seems capable of quite a bit of mischief, doesn't it? Well, being a Windows user myself, just observing the information flowing in makes me rather fearful of the potential for trouble... That said, it also made me realise we need to make things more robust and secure our foundations properly, or else it's scary.

P.S.: The footnotes are numerous and might make it a bit of a slog to read, but they're all valuable information, so do check out the original article.

This time I've leaned quite heavily on footnotes rather than a traditional reference list format, but I'm still rather undecided about which approach is best...

Which is better, everyone...?

1. 
   

    ↩

   
   



2. 
   

    ↩

   
   



3. 
   

    ↩

   
   



4. 
   

    ↩

   
   



5. 
   

    ↩

   
   



6. 
   

    ↩

   
   



7. 
   

    ↩

   
   



8. 
   

    ↩

   
   



9. 
   

    ↩

   
   



10. 
    

     ↩