From 24 Hours to 2 Hours: How We Fixed a Broken BI System With Apache Airflow

The System Was Broken, and Everyone Knew It

Our dashboards refreshed overnight. That was the expectation. Then, one week, they started taking six hours. Then eight. On a bad day, the full 24 hours. Business users would come in on Monday morning and still see Friday's numbers.

The data was wrong, too. Not wrong in an obvious way. Wrong in the quiet way where someone in finance notices a number looks off, checks it manually, finds a discrepancy, and then stops trusting the system. That is the worst kind of mistake. Because once trust is gone, you do not just have a technical problem. You have a people problem.