Pinecone vs Pgvector vs Upstash Vector DB Benchmark

In this blog post, I will share the findings of Upstash Team from a recent benchmark they ran between Upstash Vector, Pinecone, and pgvector (via Supabase).

I am sharing the benchmark article here: ( See the the full article from created by Pinecone. Originally, it was designed for benchmarking Pinecone and pgvector in local databases. We and pgvector in Supabase with the which provides great insight for both services.

Supabase also has a great blog post called . These are:

• 
  nq768: (2.6M vectors from Google Natural Questions )


• 
  cohere768: (10M vectors from Cohere-embedded Wikipedia articles)

Benchmark Logic and Specific Flags

If you are not planning to replicate these benchmarks or do not want to learn the details behind it, you can skip to the next part.

While upserting vectors into databases, we use batch operations to reduce the number of requests. However, the size of these batches is not fixed. Here is how the batch size (the number of vectors in each request) is calculated:

The code first calculates a size-based batch size based on message size constraints—specifically ensuring the total data won’t exceed the limit. It then compares this computed size-based batch size to a predefined max batch size (in this case, 1000) that might be set for other operational or performance reasons. By taking the minimum of these two values, the system ensures that both conditions are met:

1. 
   Not Exceeding the Message Size Limit: The batch fits within the message size constraint.
   
   
   
   
   1. Upstash: 10MB
   
   
   2. Pinecone: 2MB
   
   
   
   
   
   



2. 
   
   Staying Within Operational Limits: The batch doesn’t exceed a predefined threshold (defined by service providers).
   
   
   
   
   1. Upstash: 1000
   
   
   2. Pinecone: 1000
   
   
   
   
   
   

For Supabase/pgvector, we used the default chunk size of 500 in the vecs Python client.

top_k is 100 for search operations in VSB.

To replicate the benchmark yourself:

• For Upstash, you need to provide the token and the URL of your index before running the benchmark.


• For Pinecone, you need to provide an API key.


• For Supabase/pgvector, you need to provide a connection string to connect to your database.

View Upstash Command

vsb --database=upstash --workload=your_workload \

--upstash_vector_rest_url="" \

--upstash_vector_rest_token="" \

--overwrite

View Pinecone Command

vsb --database=pinecone --workload=your_workload \

--pinecone_api_key="" \

View Supabase/pgvector Command

vsb --database=supabase --workload=your_workload \

--supabase_connection_string="" \

--overwrite

We created all the vector databases in us-east-1 region and set up a GCP instance in us-central1 region to run the benchmarks.

Since Upstash Vector and Pinecone are serverless, we did not need to configure any settings while creating the databases and running the benchmarks.

For Supabase/pgvector, we used the 8XL compute instance to host the database while indexing and querying which has the following specifications:

• CPU: 32-core ARM (dedicated),


• Memory: 128 GB


• Max DB Size (Recommended): 4 TB

The choice was based on ensuring that the index fits into the maintenance_work_mem variable and for our largest workload cohere768, the index was around 40GBs. Additionally, our database size was approximately 140GBs in total which is way less than the recommended max DB size for this instance. While not running the benchmarks, we changed the compute instance type to the cheapest one (nano) to keep the cost low.

Now, let's see the results!

Results

Performance Comparison

You can scroll horizontally to view the full table.

Population is the operation of adding vectors to the database. Index creation is the operation of creating an index in the database which allows for efficient search.

Recall is a measure of how many of the top-k results are relevant to the query. The higher the recall, the more relevant the results are to the query. Recall = TP / (TP + FN) where TP = True Positives and FN = False Negatives.

Graphs

Key Takeaways

• Upstash Vector is the cheapest and offers good performance.


• Pinecone offers strong performance at a much higher price.


• pgvector in Supabase is cheap and highly configurable, but it's slower and not as easy to use as the other services.

Observations & Challenges

While benchmarking, we once again noticed how easy it is to use serverless services compared to others. In our case, we constantly needed to change our compute instance type to match the workload, all while trying to keep costs low in Supabase. This is because in pgvector, the index needs to fit in RAM; otherwise, the build throughput drops sharply. Here is a quote from the on this:

The “expansion factor” — the ratio of the index RAM to the original dataset — varies significantly across the different datasets. This is counterintuitive: there is no simple way to figure out how to size the index’s working set memory, and the consequences of getting this wrong are significant.

Additionally, deciding when to create an index in Supabase/pgvector is crucial. The index can be built before or after upserting. If you create it before upserting, the time it takes to populate the index increases significantly. In our benchmarks, it took three times longer to populate the index using the nq768 workload. However, if you create the index after upserting, the index creation may fail since it’s a resource-intensive task requiring a persistent connection to the database. Here is a useful which is the best place to start if you want to learn more about pgvector.

Conclusion

Summary: Upstash Vector, while being significantly cheaper, is slower than Pinecone. Compared to pgvector in Supabase, it is similar in cost but significantly easier to use and faster. Additionally, all the databases have good recall scores but Upstash Vector is trailing behind Pinecone and pgvector in terms of recall scores.

These benchmarks show us where we can still improve, and we are very determined to give the best experience to developers. As proof of this commitment, we recently shipped index support for Upstash Vector. We have even more exciting plans for the future.

To learn more about Upstash Vector, please check out the if you have any questions.