Ten Mistakes Marred Firewall Upgrade At Australian Telco, Contributing To Two Deaths

An independent review found that at least ten technical and process failures during a routine firewall upgrade at Australia's Optus prevented emergency calls from reaching Triple Zero for 14 hours, during which 455 calls failed and two callers died. The Register reports: On Thursday, Optus published an independent report (PDF) on the matter written by Dr Kerry Schott, an Australian executive who has held senior management roles at many of the country's most significant businesses. The report found that Optus planned 18 firewall upgrades and had executed 15 without incident. But on the 16th upgrade, Optus issued incorrect instructions to its outsourced provider Nokia. [...] Schott summarized the incident as follows: "Three issues are clear during this incident. The first is the very poor management and performance within [Optus] Networks and their contractor, Nokia. Process was not followed, and incorrect procedures were selected. Checks were inadequate, controls avoided and alerts given insufficient attention. There appeared to be reticence in seeking more experienced advice within Networks and a focus on speed and getting the task done, rather than an emphasis on doing things properly."

The review also found that Optus' call center didn't appreciate it could be "the first alert channel for Triple Zero difficulties." The document also notes that Australian telcos try to route 000 calls during outages, but that doing so is not easy and is made harder by the fact that different smartphones behave in different ways. Optus does warn customers if their devices have not been tested for their ability to connect to 000, and maintains a list of known bad devices. But the report notes Optus's process "does not capture so-called 'grey' devices that have been bought online or overseas and may not be compliant." "To have a standard firewall upgrade go so badly is inexcusable," the document states. "Execution was poor and seemed more focussed on getting things done than on being right. Supervision of both network staff and Nokia must be more disciplined to get things right."

Read more of this story at Slashdot.