RAG Series (5): Embedding Models — The Core of Semantic Understanding

Why Does Switching Embedding Models Make Such a Huge Difference?

In the first four articles, we built the RAG pipeline, tuned parameters, and mastered chunking strategies. But there's one question we haven't dived into:

After your documents are chunked, how do they become vectors?

This process is called Embedding. It transforms human-readable text into machine-computable vectors. The choice of Embedding model directly determines:

• Whether "apple" and "iPhone" are recognized as related


• Whether "database connection pool exhausted" and "Too many connections" match


• Whether Chinese idioms, technical jargon, and abbreviations are properly understood

This article explains how Embedding works, compares mainstream models, and runs a head-to-head retrieval comparison between OpenAI and BGE using real Chinese documents.

What Is Embedding?

One-Sentence Explanation

Embedding is a function that takes a piece of text and outputs a fixed-length numerical vector (e.g., 1024 dimensions). Semantically similar texts produce vectors that are close together in space.

Why Can Vectors Represent Meaning?

Imagine placing all words in a multi-dimensional space:

• "King" and "Queen" are close together


• "Apple (fruit)" and "Banana" are close together


• "Apple (company)" and "Google" are close together


• "Apple (fruit)" and "Apple (company)" are far apart

Embedding models learn these "semantic distances" through pre-training on massive text corpora. When you ask "How do I restart my iPhone?", the model knows "iPhone" relates to "Apple" (the company), not "apple" (the fruit).

Its Role in RAG

User Query → Embedding Model → Query Vector

                                          ↘

                                           Vector Similarity → Top-K Retrieval

                                          ↗

Document Chunk → Embedding Model → Document Vector (precomputed)

Embedding is the semantic bridge of RAG. Without it, retrieval is limited to keyword matching (like Ctrl+F). With it, you get semantic matching that understands synonyms, paraphrases, and context.

Mainstream Embedding Model Comparison

Model Overview

Key Metric: The MTEB Leaderboard

MTEB (Massive Text Embedding Benchmark) is the "college entrance exam" of Embedding models. It tests models on 50+ datasets across various tasks.

How to Read the MTEB Leaderboard:

1. Visit the MTEB Leaderboard
   


2. Focus on Retrieval Average — most relevant to RAG


3. Check Model Size — larger models are slower but usually more accurate

Key Findings from the Leaderboard:

• 
  English: OpenAI text-embedding-3-large dominates, but text-embedding-3-small offers exceptional value


• 
  Chinese: BGE series (especially bge-large-zh-v1.5) often outperforms OpenAI, and it's open-source and free


• 
  Multilingual: bge-m3 and Cohere embed-multilingual-v3.0 stand out

💡 Rule of Thumb: English → OpenAI, Chinese → BGE, Multilingual → bge-m3, Long Text → Cohere.

Practical: OpenAI vs BGE Retrieval Showdown on Chinese Documents

Experimental Design

We use the same Chinese technical document from Article 4 (the microservices architecture guide), generate embeddings with both OpenAI and BGE, and test retrieval quality on the same set of queries.

Code: Switching Embedding Models with One Change

LangChain's OpenAIEmbeddings class is compatible with all OpenAI-Format Embedding APIs (including SiliconFlow, Zhipu, Ollama, etc.), so switching models only requires changing a few configuration lines:

from langchain_openai import OpenAIEmbeddings



# --- Official OpenAI ---

openai_embed = OpenAIEmbeddings(

    model="text-embedding-3-small",

    api_key="sk-...",

    base_url="https://api.openai.com/v1",

)



# --- BGE (via SiliconFlow) ---

bge_embed = OpenAIEmbeddings(

    model="BAAI/bge-large-zh-v1.5",

    api_key="sk-...",  # SiliconFlow API Key

    base_url="https://api.siliconflow.cn/v1",

    chunk_size=32,     # SiliconFlow batch size limit: 32

)



# --- Use in RAG Pipeline ---

vectorstore = Chroma.from_documents(

    documents=chunks,

    embedding=bge_embed,  # Change only this line to switch models

)

Evaluation Query Set

We designed 5 queries covering different difficulty levels:

Results Comparison

Conclusion:

• For simple queries (direct keyword matches), both models perform similarly


• 
  For difficult queries (semantic reasoning required), BGE's Chinese advantage is clear, especially on synonyms and paraphrases

Cost Comparison

If you have a GPU, BGE can also be deployed locally for free (details below).

Local Deployment vs API Calls: How to Choose?

API Calls: Pros and Cons

Pros:

• Zero ops, one line of code


• Model versions auto-update


• Pay-per-use, no idle costs

Cons:

• Data leaves your domain (compliance risk for sensitive docs)


• Network latency and rate limits


• Costs accumulate with high-frequency usage

Local Deployment: Pros and Cons

Pros:

• Data never leaves your premises, absolute security


• No rate limits, ideal for high-frequency batch processing


• More economical over time (one-time GPU investment)

Cons:

• Requires GPU (BGE-large needs 4GB+ VRAM)


• Operational complexity (model downloads, version management, serving)


• Slow initial loading (model size: hundreds of MB to several GB)

Decision Tree

Is your data sensitive?

    ├─ Yes → Local Deployment (BGE or GTE)

    └─ No → Is call volume high?

              ├─ Yes → Local Deployment (saves money long-term)

              └─ No → API Calls (simpler)

                        Primarily Chinese? → BGE (SiliconFlow/Local)

                        Primarily English? → OpenAI text-embedding-3-small

Special Considerations for Chinese Embedding

1. Tokenization Differences

English Embedding models typically tokenize by spaces, but Chinese has no spaces. If a model isn't optimized for Chinese, it might understand "南京市长江大桥" as "Nanjing / Mayor / River Bridge" instead of "Nanjing City / Yangtze River Bridge".

BGE's Advantage: Specifically trained on Chinese corpora, with tokenization and semantic understanding optimized for Chinese.

2. Idioms and Colloquialisms

3. Domain Terminology

Technical documents contain extensive jargon (e.g., "Saga pattern", "Two-phase commit", "Eventual consistency"). BGE, trained on Chinese technical community data, typically understands these terms better than general English models.

Code Walkthrough: Model Switching Wrapper

To make model switching easy in your project, create a factory function:

import os

from langchain_openai import OpenAIEmbeddings





def build_embeddings(provider: str = "bge"):

    """

    Factory function: returns the appropriate Embedding model based on config.

    provider: "openai" | "bge" | "local"

    """

    if provider == "openai":

        return OpenAIEmbeddings(

            model="text-embedding-3-small",

            api_key=os.getenv("OPENAI_API_KEY"),

        )

    elif provider == "bge":

        return OpenAIEmbeddings(

            model="BAAI/bge-large-zh-v1.5",

            api_key=os.getenv("SILICONFLOW_API_KEY"),

            base_url="https://api.siliconflow.cn/v1",

            chunk_size=32,

        )

    elif provider == "local":

        # Requires: pip install sentence-transformers

        from langchain_community.embeddings import HuggingFaceEmbeddings

        return HuggingFaceEmbeddings(

            model_name="BAAI/bge-large-zh-v1.5",

            model_kwargs={"device": "cuda"},  # or "cpu"

            encode_kwargs={"normalize_embeddings": True},

        )

    else:

        raise ValueError(f"Unknown provider: {provider}")





# Usage: one line to switch

embeddings = build_embeddings("bge")  # Change this line to switch

Local BGE Deployment (Optional)

If you have a GPU, local deployment is simple:

pip install sentence-transformers

from langchain_community.embeddings import HuggingFaceEmbeddings



embeddings = HuggingFaceEmbeddings(

    model_name="BAAI/bge-large-zh-v1.5",

    model_kwargs={"device": "cuda"},

    encode_kwargs={"normalize_embeddings": True},

)



# Test

result = embeddings.embed_query("Testing Chinese Embedding")

print(f"Vector dimensions: {len(result)}")  # 1024

The first run auto-downloads the model (~1.2GB), then caches locally.

Summary and Quick Reference

Core Takeaways

1. 
   Embedding is the semantic bridge of RAG — choosing the wrong model directly hurts retrieval accuracy


2. 
   English → OpenAI, Chinese → BGE — validated by both MTEB rankings and real-world tests


3. 
   Simple queries show little difference, complex semantic queries show large gaps — BGE excels at synonyms, idioms, and terminology


4. 
   Switching models takes one line of code — LangChain's abstraction makes model swapping cost-free

Embedding Model Quick Selection Guide

References

• 
  MTEB Leaderboard — Authoritative Embedding model rankings


• 
  BGE Official GitHub — BGE series models and documentation


• SiliconFlow Embedding API


• Cohere Embed Documentation