The same internal-docs Q&A problem every company has, solved on Azure AI Search and Azure OpenAI. Terraform provisions everything, model deployments are one-line swaps in tfvars, and a small evaluation catches regressions before they hit prod.
Same problem as every company: product docs, FAQs, and policy PDFs pile up in a shared drive, support agents can't find the right answer fast enough, and the same questions land in the same Slack channel every week. This post builds the fix - internal knowledge Q&A - on Azure AI Search paired with Azure OpenAI.
Azure's pattern connects three pieces in one call: Blob Storage holds your documents, Azure AI Search indexes them with vector and keyword search, and Azure OpenAI's chat completions API with a data source parameter handles retrieval, reranking, and generation together. Terraform provisions the infrastructure. Model deployments are variables, so upgrading either the embedding model or the generation model is a tfvars change, not a rewrite. 🎯
🏗️ The Architecture
Docs (PDF/HTML/TXT) → Blob Storage
↓
Azure AI Search (integrated vectorization + indexer)
↓
Vector + keyword index (hybrid search)
↓
Azure OpenAI "On Your Data" → grounded answer + citations
Azure AI Search's integrated vectorization means the indexer calls the embedding model for you during ingestion, and again at query time for the search text. You don't write embedding code.
🔧 Terraform: The Full Pipeline
Model Deployments - Model-Agnostic by Design
# variables.tf
variable "embedding_model" {
description = "Embedding model deployment. Change this to upgrade embeddings."
type = object({
name = string
version = string
dimensions = number
})
default = {
name = "text-embedding-3-large"
version = "1"
dimensions = 1024
}
}
variable "generation_model" {
description = "Chat model deployment. Change to upgrade generation quality."
type = object({
name = string
version = string
})
default = {
name = "gpt-4.1"
version = "2025-04-14"
}
}
Azure OpenAI Deployments
# openai.tf
resource "azurerm_cognitive_account" "openai" {
name = "${var.environment}-support-openai"
resource_group_name = azurerm_resource_group.this.name
location = var.location
kind = "OpenAI"
sku_name = "S0"
}
resource "azurerm_cognitive_deployment" "embedding" {
name = "embedding-${var.embedding_model.name}"
cognitive_account_id = azurerm_cognitive_account.openai.id
model {
format = "OpenAI"
name = var.embedding_model.name
version = var.embedding_model.version
}
sku {
name = "Standard"
capacity = var.embedding_tpm
}
}
resource "azurerm_cognitive_deployment" "generation" {
name = "chat-${var.generation_model.name}"
cognitive_account_id = azurerm_cognitive_account.openai.id
model {
format = "OpenAI"
name = var.generation_model.name
version = var.generation_model.version
}
sku {
name = "GlobalStandard"
capacity = var.generation_tpm
}
}
Storage for Source Documents
# storage.tf
resource "azurerm_storage_account" "docs" {
name = "${var.environment}supportdocs"
resource_group_name = azurerm_resource_group.this.name
location = var.location
account_tier = "Standard"
account_replication_type = "LRS"
min_tls_version = "TLS1_2"
}
resource "azurerm_storage_container" "documents" {
name = "documents"
storage_account_id = azurerm_storage_account.docs.id
container_access_type = "private"
}
Azure AI Search Service
# search.tf
resource "azurerm_search_service" "this" {
name = "${var.environment}-support-search"
resource_group_name = azurerm_resource_group.this.name
location = var.location
sku = var.search_sku
identity {
type = "SystemAssigned"
}
}
Config Bridge - Terraform to SDK
# config.tf
resource "local_file" "rag_config" {
filename = "${path.module}/app/rag_config.json"
content = jsonencode({
search_endpoint = "https://${azurerm_search_service.this.name}.search.windows.net"
openai_endpoint = azurerm_cognitive_account.openai.endpoint
embedding_deployment = azurerm_cognitive_deployment.embedding.name
generation_deployment = azurerm_cognitive_deployment.generation.name
embedding_dimensions = var.embedding_model.dimensions
index_name = "support-docs-${var.embedding_model.name}-${var.embedding_model.dimensions}"
storage_container = azurerm_storage_container.documents.name
storage_account = azurerm_storage_account.docs.name
})
}
Same trick as AWS and GCP. The index name is derived from the embedding model and its dimensions. Change the model, get a new index name automatically, without touching a single downstream resource.
🐍 Index, Skillset, and Indexer (SDK)
The azurerm provider doesn't manage index schemas, skillsets, or indexers directly - those are data-plane objects created via the SDK or REST API:
# app/setup_index.py
import json
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
SearchIndex, SearchField, SearchFieldDataType,
VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile,
AzureOpenAIVectorizer, AzureOpenAIVectorizerParameters,
)
from azure.identity import DefaultAzureCredential
with open("rag_config.json") as f:
config = json.load(f)
credential = DefaultAzureCredential()
index_client = SearchIndexClient(config["search_endpoint"], credential)
vector_search = VectorSearch(
algorithms=[HnswAlgorithmConfiguration(name="hnsw-config")],
profiles=[VectorSearchProfile(
name="vector-profile",
algorithm_configuration_name="hnsw-config",
vectorizer_name="openai-vectorizer",
)],
vectorizers=[AzureOpenAIVectorizer(
vectorizer_name="openai-vectorizer",
parameters=AzureOpenAIVectorizerParameters(
resource_url=config["openai_endpoint"],
deployment_name=config["embedding_deployment"],
model_name=config["embedding_deployment"],
),
)],
)
fields = [
SearchField(name="id", type=SearchFieldDataType.String, key=True),
SearchField(name="content", type=SearchFieldDataType.String, searchable=True),
SearchField(
name="contentVector",
type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
searchable=True,
vector_search_dimensions=config["embedding_dimensions"],
vector_search_profile_name="vector-profile",
),
]
index = SearchIndex(name=config["index_name"], fields=fields, vector_search=vector_search)
index_client.create_or_update_index(index)
print(f"Index ready: {config['index_name']}")
Indexer setup (pulls from Blob Storage, chunks, and vectorizes automatically) follows the same pattern using SearchIndexerClient with a skillset that references the vectorizer. The index name carries the embedding model version, so this script is safe to re-run whenever you upgrade.
🐍 Query the Pipeline (Model-Agnostic Generation)
# app/query.py
from openai import AzureOpenAI
import json
with open("rag_config.json") as f:
config = json.load(f)
client = AzureOpenAI(
azure_endpoint=config["openai_endpoint"],
azure_ad_token_provider=get_bearer_token_provider(),
api_version="2025-01-01-preview",
)
def ask(question: str) -> dict:
response = client.chat.completions.create(
model=config["generation_deployment"],
messages=[{"role": "user", "content": question}],
extra_body={
"data_sources": [{
"type": "azure_search",
"parameters": {
"endpoint": config["search_endpoint"],
"index_name": config["index_name"],
"query_type": "vector_semantic_hybrid",
"embedding_dependency": {
"type": "deployment_name",
"deployment_name": config["embedding_deployment"],
},
},
}]
},
)
return response.choices[0].message.content
Swap generation_model.name from gpt-4.1 to gpt-4.1-mini or a future model in tfvars, terraform apply, done. No code changes. The data_sources parameter handles intent extraction, hybrid retrieval, semantic reranking, and generation in one call.
📐 Environment Configuration
# environments/dev.tfvars
environment = "dev"
search_sku = "basic"
embedding_model = { name = "text-embedding-3-large", version = "1", dimensions = 1024 }
generation_model = { name = "gpt-4.1-mini", version = "2025-04-14" }
embedding_tpm = 20
generation_tpm = 10
# environments/prod.tfvars
environment = "prod"
search_sku = "standard"
embedding_model = { name = "text-embedding-3-large", version = "1", dimensions = 1024 }
generation_model = { name = "gpt-4.1", version = "2025-04-14" }
embedding_tpm = 50
generation_tpm = 40
⚠️ The Embedding Dimension Honesty Check
Same reality as the other two clouds: a vector field's dimension count is fixed when the field is created. You cannot change vector_search_dimensions on an existing field, and you generally shouldn't mix embeddings from different models in one field even if the dimension count happens to match, since the vector spaces aren't comparable.
The index_name in the config already bakes in the embedding model and dimension count, so the upgrade pattern is the same one used for GCP:
- Change
embedding_modelintfvars
terraform apply- computes a newindex_name, doesn't touch the running index- Re-run
setup_index.py- creates the new index and indexer, re-vectorizes from the same Blob container - Run the evaluation below against both indexes
- Point
query.pyat the new index once it wins, then delete the old one
🧪 Small Evaluation: Did the Upgrade Actually Help?
# evaluate.py
from app.query import ask
eval_set = [
{"question": "What is our refund policy for annual plans?",
"expected_keywords": ["30 days", "annual", "refund", "prorate"]},
{"question": "How do I reset a customer's 2FA?",
"expected_keywords": ["admin panel", "2FA", "reset", "support ticket"]},
{"question": "What's the SLA for enterprise tier support?",
"expected_keywords": ["1 hour", "enterprise", "SLA", "priority"]},
]
def run_eval() -> dict:
hits, total_keyword_matches = 0, 0
for case in eval_set:
answer = ask(case["question"])
if answer:
hits += 1
matched = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer.lower())
total_keyword_matches += matched
return {
"answer_rate": hits / len(eval_set),
"avg_keyword_coverage": total_keyword_matches / len(eval_set),
}
# Point rag_config.json at the old index/model, run, then repoint and run again
print(run_eval())
Point the config at the old index and model first, capture the scores, then repoint at the new one and compare. Ten to twenty real questions from actual support tickets is enough to catch a regression before it reaches production.
⚠️ Gotchas and Tips
Data-plane objects live outside Terraform state. Index schemas, skillsets, and indexers are managed by the SDK, not azurerm. Terraform owns the durable infrastructure; the SDK owns content and schema.
Semantic ranking needs Standard tier. The basic SKU works for dev but semantic reranking requires standard and an explicit semantic configuration - budget for this in prod tfvars.
Shared private links only work Azure-to-Azure. If your embedding model runs outside Azure OpenAI, the indexer connection has to go over the public internet.
Re-run the indexer after content changes, not just model changes. New documents in Blob Storage don't appear in the index until the indexer runs again, either on its schedule or triggered manually.
A model-agnostic RAG pipeline for real internal questions, with an honest process for upgrading embeddings and a built-in way to check that the upgrade actually helped. Terraform for the infrastructure, SDK for the index, evaluation before every cutover. 🔍
Found this helpful? Follow for more RAG deep-dives! 💬
SOCIAL SHARE CARD GENERATOR