95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On paper, it looked like it should scale fine across the cluster. In practice, the job would race through most of its tasks and then stall. The Spark UI told the real story: almost every task finished in a...