System design interviews are changing.

Traditional questions such as “Design Twitter,” “Design Uber,” and “Design YouTube” are still important. They test whether you understand databases, caching, partitioning, replication, messaging, and high availability.

But engineers working on modern platforms now encounter a different category of problem:

• Design a ChatGPT-like conversational assistant.


• Design a retrieval-augmented generation system.


• Design an LLM inference platform.


• Design an AI agent that can call external tools.


• Design an enterprise AI assistant for private documents.


• Design an evaluation platform for generative AI applications.

These questions still require classical distributed-systems knowledge. An AI product needs APIs, queues, storage, authentication, observability, rate limiting, and reliable deployment.

The difference is that it also introduces expensive accelerators, probabilistic output, long-running requests, model routing, vector retrieval, prompt construction, safety controls, and quality evaluation.

This guide explains the most important AI system design interview questions and what a strong candidate should discuss for each.

For a broader preparation roadmap covering traditional and modern problems, see provides a structured example of this problem.

Question 2: Design a RAG System

Retrieval-augmented generation, or RAG, allows a model to answer using information retrieved from external sources.

A common interview prompt is:

Design an enterprise assistant that answers employee questions using internal documents and provides citations.

A RAG system has two major paths:

1. The ingestion path


2. The query path

The ingestion path

Documents may come from file uploads, internal wikis, cloud drives, databases, or support systems.

The ingestion pipeline performs several stages.

Document extraction

Files must be converted into usable text.

The system may need parsers for:

• PDFs


• Word documents


• Presentations


• HTML pages


• Spreadsheets


• Scanned images

The extraction process should preserve useful metadata such as titles, headings, page numbers, owners, and access permissions.

Chunking

Long documents are divided into smaller segments.

Chunks that are too large may contain irrelevant text and consume excessive context. Chunks that are too small may lose meaning.

Possible strategies include:

• Fixed token windows


• Paragraph-based chunking


• Heading-aware chunking


• Overlapping windows


• Semantic chunking

There is no universally correct chunk size. It should be tested against representative questions.

Embedding generation

Each chunk is converted into a numerical vector using an embedding model.

The embedding service should be versioned because changing models can require re-embedding the entire corpus.

Indexing

The system stores:

• Embeddings


• Original text


• Document metadata


• Access-control information


• Source location


• Embedding version


• Update timestamp

A vector index enables semantic retrieval. A traditional inverted index can support keyword retrieval. Many production systems combine both.

The query path

When a user submits a question:

1. Authenticate the user.


2. Generate an embedding for the query.


3. Retrieve candidate chunks.


4. Apply access-control filtering.


5. Rerank the candidates.


6. Select the best context.


7. Construct the prompt.


8. Generate the answer.


9. Attach citations.


10. Evaluate or log the result.

Hybrid retrieval

Semantic retrieval is useful when the query and source use different words with similar meanings.

Keyword retrieval is useful for exact terms such as:

• Product codes


• Error messages


• Names


• Dates


• Identifiers

Combining both methods often produces better coverage.

Reranking

Vector similarity may retrieve documents that are generally related but not directly useful.

A reranker can score the top candidates more accurately before they are sent to the LLM. This improves answer quality while keeping the final prompt small.

Access control

Security is one of the most important parts of enterprise RAG.

A user should never retrieve a document they are not authorized to view. Filtering after the model has already received the document is too late.

Permissions should be enforced during retrieval, with tenant and user identity included in the query path.

Freshness and deletion

The system must react when:

• A document changes.


• A document is deleted.


• Permissions change.


• A user loses access.


• A newer policy replaces an older one.

The ingestion pipeline may use event-driven updates, periodic crawling, or both.

RAG evaluation

A RAG system should separately evaluate:

• 
  Retrieval quality: Did the system find the relevant document?


• 
  Generation quality: Did the model use the retrieved context correctly?


• 
  Citation quality: Do the cited sources actually support the answer?

This separation is important. A poor answer can result from failed retrieval even when the model behaves correctly.

Question 3: Design an LLM Inference Platform

This question focuses less on the product interface and more on the infrastructure that serves models.

A possible prompt is:

Design a multi-tenant platform that serves several large language models to millions of requests.

The platform may need to support:

• Multiple model families


• Different model sizes


• Streaming generation


• Priority tiers


• Autoscaling


• Usage accounting


• Model versioning


• Regional deployment


• Fine-tuned adapters

Inference gateway

The gateway exposes a consistent API and performs:

• Authentication


• Quota enforcement


• Request validation


• Model selection


• Token-limit checks


• Admission control


• Cost estimation

Admission control is critical. Accepting unlimited work and allowing it to queue indefinitely creates poor latency and can destabilize the system.

Model registry

The registry tracks:

• Model version


• Artifact location


• Supported hardware


• Memory requirements


• Context length


• Quantization format


• Deployment status


• Safety and evaluation results

Rollouts should use immutable versions so requests and incidents can be traced to the exact model that served them.

Model placement

Loading a large model into GPU memory can take substantial time. The scheduler cannot treat models like lightweight stateless application containers.

It must decide:

• Which models remain loaded


• How many replicas each model receives


• Where fine-tuned adapters are placed


• When models should be unloaded


• How capacity is distributed across regions

Popular models may remain warm, while rarely used models may accept a cold-start delay.

Prefill and decode

LLM inference contains two different computational phases.

Prefill processes the input prompt and can often benefit from parallel computation.

Decode generates tokens sequentially and is usually memory-bandwidth intensive.

Separating or independently scheduling these phases can improve utilization, but it also adds network and orchestration complexity.

Continuous batching

Instead of waiting for a fixed group of requests to finish together, continuous batching adds and removes requests dynamically as generation progresses.

This improves GPU utilization, especially when responses have different lengths.

The scheduler must still prevent long requests from starving shorter ones.

KV cache

The key-value cache stores intermediate attention state so the model does not recompute the entire prompt for every generated token.

KV-cache management affects:

• Maximum concurrency


• Memory pressure


• Long-context support


• Prefix reuse


• Request eviction

A shared prompt prefix—such as a large system prompt—may sometimes be cached and reused across compatible requests.

Scaling

GPU utilization alone may not be sufficient for autoscaling.

Useful signals include:

• Queue length


• Time to first token


• Tokens generated per second


• KV-cache pressure


• Number of active sequences


• Predicted token demand


• Model-specific backlog

Because accelerator provisioning may be slow, the platform may need reserved capacity and predictive scaling.

Graceful degradation

When capacity is limited, the system may:

• Route to a smaller model.


• Reduce the maximum output length.


• Reject low-priority requests.


• Disable expensive features.


• Queue batch workloads.


• Move traffic to another region.


• Use a third-party model provider.

A strong interview answer discusses the quality and cost consequences of each fallback.

Question 4: Design an AI Agent Platform

An AI agent does more than produce text. It can plan a sequence of actions, call tools, observe results, update its state, and continue until a goal is completed.

A typical prompt might be:

Design an enterprise agent that can search internal documents, update tickets, send emails, and request human approval for sensitive actions.

Core components

Agent orchestrator

The orchestrator controls the execution loop:

1. Receive a goal.


2. Construct the current context.


3. Ask the model for the next action.


4. Validate the proposed action.


5. Execute the selected tool.


6. Store the result.


7. Decide whether to continue.


8. Produce the final response.

The orchestrator—not the model—should enforce hard limits such as maximum steps, timeouts, budgets, and approval requirements.

Tool registry

The tool registry describes each available capability:

• Tool name


• Purpose


• Input schema


• Required permissions


• Timeout


• Retry policy


• Risk level


• Whether human approval is required

Tool definitions should be versioned because changing their schemas can break existing agent behavior.

Tool execution service

Tool calls should run through controlled executors rather than allowing the model unrestricted access to internal systems.

The executor handles:

• Authentication


• Input validation


• Secrets


• Network policy


• Timeouts


• Retries


• Audit logging


• Output normalization

High-risk operations should use narrow, purpose-built APIs.

State and memory

Agents may need several kinds of memory.

Working memory contains the current task, observations, and intermediate steps.

Session memory preserves information during one user interaction.

Long-term memory stores information across sessions.

External memory may contain documents retrieved from databases or vector indexes.

Not everything should be stored forever. Memory needs explicit retention, privacy, and deletion policies.

Human approval

Actions such as sending payments, deleting data, publishing content, or modifying production systems should not be executed solely because a model requested them.

The agent can create a proposed action, pause its workflow, and wait for authorized approval.

The approval record should contain:

• The intended action


• The relevant parameters


• Why it was proposed


• The expected effect


• The identity of the approver


• An expiration time

Idempotency

Agents may retry actions after timeouts.

Without idempotency, a retry could send the same email twice, create duplicate tickets, or repeat a transaction.

Every state-changing tool call should include a stable execution identifier or idempotency key.

Agent-specific failure modes

A strong candidate should discuss:

• Infinite planning loops


• Repeated tool calls


• Prompt injection inside retrieved content


• Tool hallucination


• Stale observations


• Excessive cost


• Partial workflow completion


• Conflicting actions


• Unauthorized data access

The system should impose:

• Maximum step counts


• Token budgets


• Time limits


• Per-tool permissions


• Human checkpoints


• Detailed audit logs


• Recovery or compensation workflows

The introduces the core building blocks behind scalable systems, including databases, caches, queues, replication, partitioning, and load balancing.

The original is useful for practicing a consistent interview framework across modern case studies, including a complete ChatGPT design problem.

Engineers preparing for senior and staff-level discussions can continue with is a useful next step for strengthening scalability, observability, fault tolerance, and performance reasoning.

For every AI design problem, practice three times:

1. Design the happy path.


2. Design for failure and overload.


3. Defend the quality, safety, and cost trade-offs.

That third pass is where most of the valuable interview discussion occurs.

Final Takeaway

AI system design is not a replacement for traditional system design.

It is a traditional system design combined with a new set of constraints.

You still need to understand APIs, storage, caching, queues, partitioning, replication, security, observability, and fault tolerance.

But you must now apply those concepts to systems with:

• Probabilistic outputs


• Expensive inference


• Streaming generation


• Vector retrieval


• Dynamic prompts


• Long-lived context


• External tools


• Model evaluation


• Safety requirements


• Human approval

Start with four foundational problems:

1. Design ChatGPT.


2. Design a RAG platform.


3. Design an LLM inference service.


4. Design an AI agent platform.

Master the request flow, deep dives, failure modes, and trade-offs behind each one.

Once you can explain those systems clearly, most other AI system design questions become variations of the same underlying building blocks.