Why AI Systems Become Expensive: Tokenization, Chunking, and Retrieval Design in the Cloud (AWS)

When building modern AI knowledge systems, discussions often jump directly to prompts, retrieval pipelines, or model selection. However, long before a model generates an answer, something more fundamental happens that your data must be transformed into a format that models can understand and retrieve efficiently.

This transformation typically involves several foundational steps:

1. Tokenization – Converting raw text into model-readable units

2. Chunking – Splitting documents into manageable segments

3. Vectorization – Converting text into embeddings

4. Indexing – Storing vectors for efficient similarity search

These steps form the foundation of retrieval-based AI systems, and design decisions at this stage often have a greater impact on system performance than prompt engineering or model tuning.

These architectural considerations are also increasingly relevant for modern AI development tools such as Claude Code, OpenAI Codex–based systems, and other AI-powered coding assistants. Although these tools may appear to operate purely through conversational prompts, they often rely on similar retrieval pipelines under the hood, indexing codebases, documentation, and contextual information before generating responses. As a result, the same factors discussed in this article, such as tokenization, chunking strategies, embeddings, and retrieval design, can significantly influence both the performance and the token consumption of these systems.

In this article, we explore how these processes work together and how they can be implemented using Amazon Bedrock and Amazon OpenSearch Service.

The Retrieval Pipeline Begins Before Retrieval

Before a model retrieves or generates information, data must first pass through a preparation pipeline.

Each stage introduces trade-offs affecting:

• Retrieval accuracy


• Latency


• Operational cost

Designing this pipeline carefully is essential for scalable GenAI systems.

Tokenization: How Models Read Text

Large language models do not process text directly. Instead, they operate on tokens, which are smaller units derived from text.

Tokens may represent:

• Whole words


• Parts of words


• Punctuation


• Whitespace

For example:

Cloud computing enables scalable AI systems.

might be tokenized as:

["Cloud", " computing", " enables", " scalable", " AI", " systems", "."]

Tokenization is necessary because models operate within fixed context windows, defined by the maximum number of tokens they can process at once.

However, token counts vary depending on the model's tokenizer.

Tokenization Algorithms

Tokenization is performed using trained algorithms rather than simple whitespace splitting.

Common approaches include:

Many modern models rely on Byte Pair Encoding or similar subword tokenization techniques (Sennrich et al., 2016).

Example: Byte Pair Encoding

Consider the word: tokenization, A BPE tokenizer may split it as:

["token", "ization"] or ["token", "iz", "ation"] depending on how the tokenizer vocabulary was learned.

Subword tokenization allows models to represent rare or unseen words efficiently without requiring extremely large vocabularies.

Why Tokenization Matters

Tokenization affects several key aspects of AI system design.

Context Window Limits

Models can only process a limited number of tokens at once. Depending on the model, context windows may range from thousands to hundreds of thousands of tokens.

Large documents therefore need to be split into smaller chunks before processing.

Cost

Many generative AI platforms charge based on token usage. When generating embeddings or performing inference with can assist in identifying semantic boundaries.

This often produces more meaningful chunks, improving retrieval quality.

Vectorization: Converting Text into Embeddings

After chunking, each segment is converted into a vector embedding.

Embeddings represent text as numerical vectors capturing semantic meaning.

Example:

"cloud infrastructure"

might become:

[0.43, 0.24, 0.54, 0.12, 0.53]

Embeddings allow systems to compare semantic similarity between text segments.

Embedding generation can be performed using models provided through Amazon Bedrock.

Optimizing Embeddings

Embedding design has a direct impact on performance and cost.

Embeddings vary in Vector Dimensionality, and higher-dimensional vectors can capture more semantic nuance but require more storage and computational resources.

Indexing Embeddings for Retrieval

Once embeddings are generated, they must be stored in a structure supporting efficient similarity search.

Vector databases achieve this using Approximate Nearest Neighbor (ANN) algorithms, which allow searching across millions of vectors efficiently (Aumüller et al., 2020).

Amazon OpenSearch Service supports vector search capabilities using ANN indexing.

Typical workflow:

Example AWS Architecture

This architecture separates concerns across:

• Ingestion


• Processing


• Embedding generation


• Vector indexing


• Retrieval

Example: Generating Embeddings with AWS Bedrock

import boto3

import json



bedrock = boto3.client("bedrock-runtime")



text = "Cloud architecture improves scalability and resilience."



response = bedrock.invoke_model(

    modelId="amazon.titan-embed-text-v1",

    body=json.dumps({"inputText": text})

)



embedding = json.loads(response['body'].read())



print(embedding)

The generated vector can then be indexed in Amazon OpenSearch Service for similarity search.

Final Thoughts

• Tokenization, chunking, vectorization, and indexing form the core infrastructure of retrieval-based AI systems.


• Although much attention is given to models and prompt engineering, the effectiveness of an AI system often depends more on how knowledge is structured and indexed before retrieval.


• Platforms such as Amazon Bedrock and Amazon OpenSearch Service provide powerful building blocks for implementing these pipelines.


• However, achieving high-quality retrieval ultimately depends on designing the data preparation process thoughtfully


• Efficient tokenization and chunking strategies not only improve retrieval quality but also play a critical role in controlling token consumption and operational cost in large-scale GenAI systems.

References

• Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL.


• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.


• Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.


• Aumüller, M., Bernhardsson, E., & Faithfull, A. (2020). ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms.


• Salton, G., & Buckley, C. (1988). Term-Weighting Approaches in Automatic Text Retrieval.


• AWS Documentation – Amazon Bedrock


• AWS Documentation – Amazon OpenSearch Vector Search

If you're building AI systems today, optimizing tokenization and chunking strategies early can save significant cost and complexity later.