The same internal-docs Q&A problem every company has, solved on Vertex AI RAG Engine. Terraform provisions the infrastructure, the SDK manages the corpus, and the design keeps your generation model swappable and your embedding upgrades painless.
Same problem as every company: product docs, FAQs, and policy PDFs pile up, support agents can't find the right answer fast enough, and the same questions land in the same Slack channel every week. This post builds the fix - internal knowledge Q&A - on Vertex AI RAG Engine, GCP's managed RAG service. RAG Engine handles chunking, embedding, and retrieval internally with a managed vector database.
Terraform provisions the infrastructure that changes rarely: APIs, the GCS bucket, IAM, and the RAG Engine's database tier. The Python SDK manages the corpus and file operations - the things that change often. The design keeps your generation model swappable with a one-line change, and gives you an honest, workable pattern for upgrading embedding models too. 🎯
🏗️ The Architecture
Docs (PDF/HTML/TXT) → GCS bucket
↓
RAG Engine Corpus (chunking + embedding)
↓
RagManagedDb (vector storage)
↓
retrieveContexts + generateContent → grounded answer
🔧 Terraform: Infrastructure Layer
APIs and Config
# rag/main.tf
resource "google_project_service" "required" {
for_each = toset([
"aiplatform.googleapis.com",
"storage.googleapis.com",
])
project = var.project_id
service = each.value
}
resource "google_vertex_ai_rag_engine_config" "this" {
region = var.region
rag_managed_db_config {
dynamic "basic" {
for_each = var.rag_db_tier == "basic" ? [1] : []
content {}
}
dynamic "scaled" {
for_each = var.rag_db_tier == "scaled" ? [1] : []
content {}
}
}
depends_on = [google_project_service.required]
}
RAG Engine lets you scale your RagManagedDb instance using a choice of tiers: Basic, Scaled, and Unprovisioned. The tier is a project-level setting that impacts all RAG corpora using RagManagedDb. Use basic for dev, scaled for prod traffic.
GCS Bucket for Source Documents
# rag/storage.tf
resource "google_storage_bucket" "docs" {
name = "${var.project_id}-${var.environment}-support-docs"
location = var.region
uniform_bucket_level_access = true
versioning {
enabled = true
}
}
Service Account and IAM
# rag/iam.tf
resource "google_service_account" "rag_app" {
account_id = "${var.environment}-rag-app"
display_name = "RAG Application Service Account"
project = var.project_id
}
resource "google_project_iam_member" "aiplatform_user" {
project = var.project_id
role = "roles/aiplatform.user"
member = "serviceAccount:${google_service_account.rag_app.email}"
}
resource "google_storage_bucket_iam_member" "docs_reader" {
bucket = google_storage_bucket.docs.name
role = "roles/storage.objectViewer"
member = "serviceAccount:${google_service_account.rag_app.email}"
}
Model Config - Bridge to the SDK
# rag/config.tf
resource "local_file" "rag_config" {
filename = "${path.module}/app/rag_config.json"
content = jsonencode({
project_id = var.project_id
region = var.region
gcs_bucket = google_storage_bucket.docs.name
embedding_model = var.embedding_model
generation_model_id = var.generation_model_id
corpus_display_name = "${var.environment}-support-docs-${replace(var.embedding_model, "/[^a-zA-Z0-9]/", "-")}"
chunk_size = var.chunk_size
chunk_overlap = var.chunk_overlap
})
}
This is the key design decision. The corpus display name is derived from the embedding model. Change the embedding model variable, and Terraform computes a new corpus name automatically - setting up the upgrade pattern covered below.
🐍 Corpus Creation and Ingestion (SDK)
# app/setup_corpus.py
import json
from vertexai import rag
import vertexai
with open("rag_config.json") as f:
config = json.load(f)
vertexai.init(project=config["project_id"], location=config["region"])
embedding_model_config = rag.RagEmbeddingModelConfig(
publisher_model=f"publishers/google/models/{config['embedding_model']}"
)
corpus = rag.create_corpus(
display_name=config["corpus_display_name"],
rag_embedding_model_config=embedding_model_config,
)
rag.import_files(
corpus_name=corpus.name,
paths=[f"gs://{config['gcs_bucket']}/"],
transformation_config=rag.TransformationConfig(
chunking_config=rag.ChunkingConfig(
chunk_size=config["chunk_size"],
chunk_overlap=config["chunk_overlap"],
)
),
max_embedding_requests_per_min=900,
)
print(f"Corpus ready: {corpus.name}")
🐍 Query the Pipeline (Model-Agnostic Generation)
# app/query.py
from vertexai import rag
from vertexai.generative_models import GenerativeModel, Tool
def ask(question: str, corpus_name: str, generation_model_id: str) -> str:
rag_tool = Tool.from_retrieval(
retrieval=rag.Retrieval(
source=rag.VertexRagStore(
rag_resources=[rag.RagResource(rag_corpus=corpus_name)],
similarity_top_k=5,
)
)
)
model = GenerativeModel(generation_model_id, tools=[rag_tool])
response = model.generate_content(question)
return response.text
answer = ask(
"What is our refund policy for annual plans?",
corpus_name="projects/.../ragCorpora/...",
generation_model_id="gemini-2.5-flash",
)
The generation model is fully swappable at query time. Swap gemini-2.5-flash for gemini-2.5-pro, or even a Model Garden model like Llama, by changing generation_model_id in tfvars. No corpus changes needed.
⚠️ The Embedding Model Honesty Check
Here's the part most tutorials skip: the embedding model is locked to the corpus at creation time. You cannot change it without recreating the corpus and re-importing all data. The association between your embedding model and the RAG corpus remains fixed for the lifetime of that corpus.
This is why the Terraform config above names the corpus after the embedding model. Upgrading embeddings means standing up a new corpus, not editing the old one. The pattern:
- Change
embedding_modelintfvars(e.g.,text-embedding-004→text-embedding-005)
terraform apply- this computes a newcorpus_display_name, nothing destructive happens to the old corpus- Run
setup_corpus.pyagain - creates a fresh corpus under the new name, re-imports from the same GCS bucket - Run the evaluation below against both corpora
- If the new corpus wins, point your app config at the new
corpus_nameand delete the old corpus
This is genuinely simple once you know the pattern, and it means zero downtime during the swap since both corpora exist side by side until you cut over.
📐 Environment Configuration
# environments/dev.tfvars
environment = "dev"
embedding_model = "text-embedding-005"
generation_model_id = "gemini-2.5-flash"
rag_db_tier = "basic"
chunk_size = 512
chunk_overlap = 100
# environments/prod.tfvars
environment = "prod"
embedding_model = "text-embedding-005"
generation_model_id = "gemini-2.5-pro"
rag_db_tier = "scaled"
chunk_size = 512
chunk_overlap = 100
Chunking configuration is set per import operation, not per corpus, so you can re-import the same files with different chunking to test what works best, without even touching the embedding model.
🧪 Small Evaluation: Did the Upgrade Actually Help?
Same lightweight approach as any model regression check - a handful of real questions, scored on retrieval hit rate and answer keyword coverage:
# evaluate.py
from vertexai import rag
from app.query import ask
eval_set = [
{"question": "What is our refund policy for annual plans?",
"expected_keywords": ["30 days", "annual", "refund", "prorate"]},
{"question": "How do I reset a customer's 2FA?",
"expected_keywords": ["admin panel", "2FA", "reset", "support ticket"]},
{"question": "What's the SLA for enterprise tier support?",
"expected_keywords": ["1 hour", "enterprise", "SLA", "priority"]},
]
def run_eval(corpus_name: str, generation_model_id: str) -> dict:
hits, total_keyword_matches = 0, 0
for case in eval_set:
contexts = rag.retrieval_query(
rag_resources=[rag.RagResource(rag_corpus=corpus_name)],
text=case["question"],
rag_retrieval_config=rag.RagRetrievalConfig(top_k=5),
)
if contexts.contexts.contexts:
hits += 1
answer = ask(case["question"], corpus_name, generation_model_id).lower()
matched = sum(1 for kw in case["expected_keywords"] if kw.lower() in answer)
total_keyword_matches += matched
return {
"retrieval_hit_rate": hits / len(eval_set),
"avg_keyword_coverage": total_keyword_matches / len(eval_set),
}
old_results = run_eval("CORPUS_004_NAME", "gemini-2.5-flash")
new_results = run_eval("CORPUS_005_NAME", "gemini-2.5-flash")
print(f"text-embedding-004 corpus: {old_results}")
print(f"text-embedding-005 corpus: {new_results}")
Run this before cutting over. If the new embedding model's retrieval_hit_rate drops, don't delete the old corpus yet - investigate why. Ten to twenty real questions pulled from actual support tickets catch regressions well enough for this kind of sanity check.
⚠️ Gotchas and Tips
Corpus creation and import happen via SDK only. There's no native Terraform resource for the RAG corpus itself as of this writing. Terraform handles the durable infrastructure; the SDK handles corpus lifecycle.
Retrieval rate limit. Retrieval requests are capped at 600 RPM per region. Factor this into evaluation batch sizes and production query volume.
Delete old corpora deliberately, not automatically. Since corpora are created outside Terraform state, cleanup is a manual or scripted step, not something terraform destroy handles for you.
Watch for model deprecations. Google occasionally deprecates publisher embedding models. Since the corpus is locked to whatever model created it, plan a migration path before a deprecation date arrives, not after.
A model-agnostic RAG pipeline for real internal questions, with an honest process for upgrading embeddings and a built-in way to check that the upgrade actually helped. Terraform for the infrastructure, SDK for the corpus, evaluation before every cutover. 🔍
Found this helpful? Follow for the Azure version coming next! 💬
SOCIAL SHARE CARD GENERATOR