Most "RAG tutorials" stop at a single embedding query against a single index. That works for a demo and falls over the moment a real user asks something like "compare our Q3 and Q4 vendor contracts and flag anything that changed" — a question that needs multiple sub-queries, reasoning about what's still missing, and synthesis across documents.
Microsoft Foundry's answer to this is Foundry IQ: an agentic retrieval layer built on Azure AI Search that treats retrieval as a reasoning task rather than a single keyword or vector lookup. This is a from-scratch build of a semantic search pipeline using GPT-5 for query planning/synthesis and Foundry IQ for retrieval.
Architecture
Instead of one query hitting one index once, Foundry IQ's knowledge base plans sub-queries, executes them in parallel against one or more knowledge sources, evaluates whether it has enough signal, and iterates before synthesizing a final, cited answer.
Step 1: Create a knowledge source
A knowledge source is a reusable reference to your underlying content — in this example, a Blob Storage container of documents. Creating it also triggers Azure AI Search to generate the index, skillset, and indexer needed to chunk and vectorize the content automatically.
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
SearchIndexKnowledgeSource,
SearchIndexKnowledgeSourceParameters,
)
from azure.identity import DefaultAzureCredential
index_client = SearchIndexClient(
endpoint="https://<your-search-service>.search.windows.net",
credential=DefaultAzureCredential(),
)
knowledge_source = SearchIndexKnowledgeSource(
name="vendor-contracts-ks",
search_index_parameters=SearchIndexKnowledgeSourceParameters(
search_index_name="vendor-contracts-index",
),
)
index_client.create_or_update_knowledge_source(knowledge_source=knowledge_source)
Step 2: Create a knowledge base with GPT-5 for planning and synthesis
The knowledge base ties one or more knowledge sources together with an LLM deployment that handles query planning and answer synthesis.
from azure.search.documents.indexes.models import (
KnowledgeBase,
KnowledgeBaseAzureOpenAIModel,
AzureOpenAIVectorizerParameters,
)
knowledge_base = KnowledgeBase(
name="vendor-contracts-kb",
knowledge_sources=[{"name": "vendor-contracts-ks"}],
models=[
KnowledgeBaseAzureOpenAIModel(
azure_open_ai_parameters=AzureOpenAIVectorizerParameters(
resource_url="https://<your-foundry-resource>.openai.azure.com",
deployment_name="gpt-5-mini",
model_name="gpt-5-mini",
)
)
],
output_configuration={
"modality": "answerSynthesis", # verbatim extractive data is the alternative
},
)
index_client.create_or_update_knowledge_base(knowledge_base=knowledge_base)
output_configuration is the key lever here: answerSynthesis returns a pre-generated, cited answer, while extractive mode returns verbatim source chunks and leaves reasoning entirely to your agent's own model. Extractive mode costs less and gives your agent more control; synthesis mode does more work up front at the retrieval layer.
Step 3: Wire the knowledge base into a Foundry agent via MCP
Each knowledge base exposes a standalone MCP endpoint. Any MCP-compatible client — including Foundry Agent Service, but also GitHub Copilot or other MCP clients — can call its knowledge_base_retrieve tool directly.
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
project = AIProjectClient(
endpoint="https://<your-foundry-project-endpoint>",
credential=DefaultAzureCredential(),
)
agent = project.agents.create_agent(
model="gpt-5-mini",
name="contract-search-agent",
instructions="Answer questions about vendor contracts using the knowledge base tool. Always cite sources.",
tools=[{
"type": "mcp",
"server_url": "https://<your-search-service>.search.windows.net/knowledgebases/vendor-contracts-kb/mcp?api-version=2026-05-01-preview",
"allowed_tools": ["knowledge_base_retrieve"],
}],
)
Reasoning effort and cost/latency tradeoffs
The knowledge base's retrieval_reasoning_effort setting controls how much LLM-driven planning happens before retrieval, and it's the main dial for balancing latency, cost, and answer quality.
| Reasoning effort | What happens | Best for |
|---|---|---|
| Minimal | Bypasses LLM query planning entirely; direct hybrid search | Simple factual lookups, latency-sensitive paths |
| Medium | LLM reformulates and may decompose the query into sub-queries | Multi-part or ambiguous questions |
| High | Full iterative planning, evaluates sufficiency, re-queries as needed | Complex, multi-hop questions across many documents |
Lowering reasoning effort is also the primary way to control the number of GPT-5 tokens consumed per query — fewer planning passes and less iteration directly reduce both latency and inference cost, without needing to touch the underlying index.
Multi-hop queries in practice
The reason this matters versus classic RAG: a query like "which vendors had payment terms that changed between the Q3 and Q4 renewals" can't be answered by a single embedding lookup. With medium or high reasoning effort, the knowledge base decomposes it into sub-queries (find Q3 renewals, find Q4 renewals, compare payment terms fields), runs them in parallel against the knowledge source, and only synthesizes a final answer once it judges the retrieved context sufficient — re-querying automatically if it isn't.
SOCIAL SHARE CARD GENERATOR