Transparency in benchmarking

Written by Ben Dicken

Database benchmarks are imperfect. They are also useful.

No benchmark can tell you exactly how a database will perform for your application. Workload shape, data size, region placement, storage, configuration, and cost all matter. But fair benchmarks help customers understand tradeoffs, compare options, and ask better questions before choosing infrastructure.

The DeWitt clause

Many cloud vendors include language in their terms that restricts comparative benchmarking. These restrictions are called (AUP). Recently, we removed this in favor of a more open "Benchmarking" section in our AUP.

The new section reads:

You may perform benchmark tests (“Benchmark”) of the Services, provided that the Benchmark is conducted in good faith and uses a fair and transparent methodology. Please refer to PlanetScale's published . These practices come from our deep experience benchmarking databases in the cloud where topology, server location, region, workload, and instance type differences materially impact the result.

We encourage other vendors, analysts, and practitioners to use the same standard. Benchmarks should be deep, thorough, technically sound, and transparent enough for others to understand and reproduce.

Our ask

We invite other vendors to adopt this same language and standard in their own AUPs. Allow public benchmarking, remove DeWitt clauses, and hold benchmarks to clear expectations for fairness and transparency.

Customers should be able to compare the systems they rely on.

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen