Chater 4 - Opening Case - Our Biggest Infrastructure Failure Flashcards
What happened on July 8, 2022 with Rogers in Canada?
The message service interruption was what people received through land line, cellular, and internet based services. There was no communication available with police, hospitals, 911, financial services, POS and telephone systems were down. There was a total disruption.Important treatments like cancer treatment had to be cancelled or redirected, concerts cancelled, and regulators like the Canadian Radio Television and Telecommunications (CRTC) could not be contacted.
What did Verges report find in Canada? What did the denial of service mitigation company (CloudFlare) detect? When did the networks improve?
The internet traffic in Canada was down 25% overall. The Rogers traffic went down to 0% at 3am. The networks only improved by 9:52pm that night , some were running in the afternoon, but many took a few days or weeks to stabilize.
What was the cause of the outage at Rogers?
A maintenance update was the cause of the failure. The distribution routers in the Rogers network had been updated. The three routers had a software error that caused too many routers to be calculated (over 900,000) since a routing filter had been deleted, thus they crashed. Multiple gateways as well as distribution routers stopped communicating with each other and all the Rogers network crashed.
What happened with Rogers in April 2021?
Rogers mobile networks were halted due to a vendor software update. All mobile services had crashed. Since only mobile systems went down, emergency services were not affected as core systems continued to run.
When everything stops how do you find the problem?
It is a technical search that highly qualified employees must partake in. You must track all hardware and software changes even if mundane. You must reverse the changes, specify the problem, or retest and debug every change going backwards from the present.
Which three companies control the data communications infrastructure in Canada? What percentage do they hold? What has the Canadian government demanded from them?
Telus, Rogers, and Bell own 90% of it, and the Canadian government has demanded that they back each other up in the event of a failure of essential services.
What did the Federal Communications Commission release two days before the shutdown?
Report and Order and Further Notice of Proposed RuleMaking on the topic of improved readiness and reliability in the event of an emergency. The idea was to provide access to roaming, have multiple providers support each other, and make consumers/local governments / others aware of what to do in an emergency.
What did the CRTC require from Rogers after the failure? What did Rogers set up?
Specify the safeguards that they would establish to enable critical functions to con tin even if there is another network failure. Through the third party advisors the company set up alternative communications carriers into critical locations and was giving key employees a second mobile methods of communicating. Management actions included an examination of processes around network reliability and automated incident management.
What was signed by Francois Philippe Champagne the federal Minister of Innovation, Science, and Industry?
A memorandum of understanding on telecommunications reliability on September 7, 2022 with 12 telecom companies. Its principles state that the companies will inform the public and government authorities about network outages based upon action plans that each telecom has prepped, and that the companies have sufficient backup plans to do those communications. This includes having available roaming services and assisting people so they have basic telephone services and can process transactions. The Minister demanded that the telecom CEO’s make them happen.
What was the Telecom reliability agenda?
Targets to improve the telecommunications infrastructure reliability. Requested that the CRTC address the issue and that the Canadian Security Telecommunications Advisory committee also report on the issue.
What happened in September and October 2022 with respect to the memorendum of understanding.
The readiness and reliability is put into question. Hurricane Fiona hit PEI causing a storm surge that knocked out power for 80% of PEI residents, half of them got their power back a week later, and more time after that. The provincial government sent out emergency services to try and help them. Now the concern is whether the telecommunications company can get their part of the deal done.
