Inside the World of Data Centers

The computing requirements of algorithms have increased dramatically over the past two decades. In particular, machine learning (ML) algorithms have experienced a growth in computing resource demand that exceeds Moore’s Law. While Moore's Law predicts a doubling of processing power every two years, since 2012, ML algorithms have been doubling in computational demands every 3-4 months (“AI and Compute,” 2018).

As a result, running these algorithms on a single computer is nearly impossible or prohibitively expensive. A more practical approach is to break down these algorithms into smaller chunks, and then use many commodity computers to run these smaller blocks. To illustrate this, imagine we are training a machine learning model on a dataset with 1 million entries. Instead of using one computer to process the entire dataset, we could break it into 10 blocks of 100K entries each. We would then use 10 computers, each running the training algorithm on the subset of 100K entries. (Note: For simplicity, I've omitted the step of combining the results from these machines, as it’s beyond the scope of this article.)