TACOMA — Thousands of emergency calls made in Washington state during the recent 911 outage failed to go through because they got stuck at a processing center in Englewood, Colo.
The estimated 4,500 calls — which failed to reach local emergency centers for about six hours April 10 — were stranded because the Colorado complex that routes 911 calls to the appropriate places had reached its threshold, according to an April 24 report to the state.
The calls initially go to the complex as part of the state’s agreement with Louisiana-based telecommunications company CenturyLink to provide a statewide 911 network, as opposed to city- and county-run systems. CenturyLink sent the report explaining the breakdown to the state Utilities and Transportation Commission, which is investigating the outage.
CenturyLink uses the company Intrado as a third-party vendor to route 911 calls to the appropriate dispatch centers. To make that happen, Intrado’s Colorado complex assigns each call a unique identifying “key,” which allows the call to be routed to the appropriate dispatchers.
In effect, the keys are tickets that let the calls be sent on their way.
On April 10, the Colorado center ran out of those tickets.
To avoid that, every so often Intrado purges the system to make sure enough keys are available for incoming calls. The last time the system was cleared out was September, said state E911 IT project manager Andy Leneweaver, based on his conversations with the company.
Between then and April 10, the 40 million to 60 million keys available were used up, meaning that during the outage incoming calls effectively were unable to leave the Colorado complex, he said.
Another factor that added to the delay in sending 911 calls to Washington’s emergency centers was that the possibility of running out of keys was not thought serious enough to trigger a significant alarm notifying Intrado of the breakdown, Leneweaver said.
“Unfortunately, this had been relegated to a relatively minor alarm condition,” he said. “It took our (call centers) calling for them to pay attention to the alarm.”
Washington’s 911 system isn’t the only one Intrado serves. The state report states 911 calls in Minnesota and North Carolina were part of the April 10 outage, though it didn’t specify how many calls.
“I think the number was six to eight (call centers) in North Carolina and somewhere in the neighborhood of 11 in Minnesota,” Leneweaver said, compared to 127 call centers affected in Washington.
Intrado has another complex in Miami that functioned just fine on April 10. The 770 Washington 911 calls that got through during the outage presumably were routed through Florida, instead of Colorado, Leneweaver said.
Once Intrado crews figured out what was causing the outage, all calls were temporarily sent to Miami as a fix, Leneweaver said.
Intrado says the April 10 outage “was caused by an isolated system issue that was promptly resolved.”
“The system is stable and continues to process 911 calls normally,” the company said in a statement after the incident.
As for the future, one thing the state has asked in light of the outage is that emergency calls be more evenly distributed between the Miami and Colorado centers, Leneweaver said.
“We want a nice, balanced network,” he said. “Because you don’t want any one single point of failure to take it down. Obviously, this was sort of a single point of failure.”
Intrado also has created a specific, enhanced alarm for when the unique key threshold is reached, according to the state report.
But the other fixes by Intrado make the chances of that happening slim, Leneweaver said. Instead of 40 million to 60 million, the threshold was increased after the outage to 6 billion to 8 billion unique keys.
“No. 2, they have a guy watching it now, actively watching it now, to see how it’s logging,” he said.
CenturyLink, in its report to the state, also seemed confident the problem would not repeat itself.
“It is not theoretically possible to exhaust the threshold ranges,” the company wrote.
