I learned one of my most important distributed-systems lessons the hard way.

We were working on a payment flow connected to an external payment gateway. On paper, the architecture looked solid: microservices, clean database transactions, retry logic, monitoring, and enough security checks to make us feel safe before deployment.

Then production...