EXCERPTED FROM
MARILYN: A WOMAN IN CHARGE by DICK MARTIN
C H A P T E R T E N
Network Down
AT&T’s 4,200 square foot Network Operations Center—built into a hillside in Bedminster, New Jersey—is normally as quiet as a college library. A two-story, curving wall of giant video screens display traffic coursing across the company’s worldwide network. But the real action happens at three rows of consoles spread across the room in front of the screens where engineers monitor more discrete elements of the network, including the signaling system that finds the most efficient route for each call and data packet, the 114 electronic switches scattered across the country that put calls and data on those routes, and all the cables over which the calls and data flow, zigzagging across the country and under the seas. The lighting is low to make it easier to read the screens. Just about the only sound is the clicking of computer keys and hushed telephone conversations.
On Monday, January 15, 1990, the network was projected to handle 115 million calls, average for a business day back then. But at 2:25 p.m. the engineers in the center began to notice an alarming number of red warning signals from various parts of the network. Within seconds, the wall of video screens was crisscrossed with a tangle of red lines as a rapidly-spreading malfunction leapfrogged from one switching center to another. While his engineers tried to figure out what was going on, the network operations manager on duty grabbed a phone and started a cascade of notifications. The long distance network was down.
One of the people notified was Marilyn Laurie. She had a PR manager assigned to the Network Operations Center. He called and told her what he knew, basically that AT&T customers couldn’t make long distance calls. At least, not on AT&T’s network. It was so inconceivable that she had trouble understanding what he was saying. “You mean some farmer tried to bury a cow and cut a line somewhere?” she said. “No, it’s not a cable cut. They don’t know what it is. But the network—the whole network—is down.” Then he excused himself to join an operations conference call to see what more he could learn. Marilyn rushed up the circular staircase in the executive offices to make sure Bob Allen knew.
When she got to his office, he was on a conference call with the senior officer responsible for the network and his counterpart at Bell Labs. She would spend the rest of the day shuttling between conference calls with the operations people, salespeople, and her own PR staff. In the process, she scrawled a long column of notes to herself on the pad of lined paper she carried with her everywhere.
Get this behind us.
Explain what happened (technical backgrounder).
Make clear problem fixed.
Give customers something.
Don’t promise won’t happen again.
Steps to minimize likelihood.
REA? News conference at NOC?[1]
Customer, opinion leader contacts, consumer leaders?
Employee letter, advertisement.
Strategy for all future communications (customer seminars, technical papers, media interviews).
The first media calls came within 35 minutes of the Network Operations Center’s alert. Her public relations people already knew the drill—acknowledge the problem, assume responsibility, volunteer the facts, minimize speculation, and correct inaccurate information. Tell the media what the company is doing to fix the problem and keep them updated as the situation changes. If there’s something you don’t know, admit it; tell the reporters you’ll try to find out; and as soon as you do know, fill them in. Don’t speculate about anything but try to anticipate what speculation will bubble to the surface so you can correct it quickly. For example, AT&T’s media relations people assumed someone would ask if the problem was the work of a hacker. An early call to Bell Labs quickly determined that wasn’t the case, and the information was put into a Q&A distributed to AT&T PR people for use if asked. Within minutes, it was asked and answered.
AT&T had a long history of dealing with the media as transparently as humanly possible. Every reporter covering the company had the office, home, and cell phone number of at least one media relations person. And contact information appeared at the top of every news release. AT&T media relations people were expected to be available 24/7. They were forbidden to offer a curt “no comment;” every question was to be answered even if only to say, “As a matter of long-standing policy, we don’t comment on rumors of material matters because if we do, we have an obligation to update you if the situation changes.” And if a story included “the company did not immediately respond for comment,” someone would have to explain why.
These policies dated back to the company’s first vice president of public relations, Arthur W. Page, who was the son of one of the founders of Doubleday Page Publishing and, in his own right, editor-in-chief of The World’s Work, one of the leading magazines covering business and public affairs in the 1920s. Thanks to Page, AT&T’s public relations people had an unusual degree of autonomy in responding to media inquiries. They were close enough to the business to know when they needed to check with the lawyers or engineers, but most of the time, they knew how to respond. Marilyn had organized her organization so public relations people were working shoulder to shoulder with the company’s line managers to identify risks—even the “unimaginable” possibility that parts of the phone network would go down. So when the unimaginable happened, line management and PR staff knew what to do.
And Marilyn didn’t need to immerse herself in the day-to-day give-and-take of media relations. She could concentrate on the bigger picture, which in this case meant figuring out what to do when the network was restored or if the problem lasted into the next day. But first she had to deal with some company executives who were more worried about their bottom line than their customers. Back then, AT&T handled about 70 percent of all long distance calls. Competitors like MCI and Sprint handled the rest. But if an AT&T customer wanted to use a competitor’s network they couldn’t just dial “1” before the number they were calling, they had to dial a ten-digit access code. Still, it was better than nothing and some customers were calling AT&T operators for the other companies’ codes. Someone in the five-to-six-level hierarchy above the operators didn’t want them to cough up the codes. While she was trying to think beyond the immediate crisis, Marilyn found herself in pitched battle with the businesspeople to reverse that decision so AT&T’s customers could make calls on its competitors’ networks until its own was restored.
Meanwhile, hundreds of engineers were frantically poring through the millions of lines of software code on which the giant switches run. In late evening, they found it—a single line of code in software installed a month earlier made the switches act as if they had reached the limit on the number of calls they could handle. Busy signals cascaded through each of the 114 switches, blocking more than 50 million calls in the nine hours the network was down. It was 10:30 p.m. before the company’s network engineers could stabilize the network by shutting off the signaling network that was propagating the error.
Many would have put out a news release and called it a day. But Marilyn argued that more was called for. She had an unusual talent for pulling back and looking at issues from a much broader perspective than those around her. To her, the network outage didn’t mean people couldn’t make phone calls, it meant they had lost something they thought they could count on. “When people get a wrong number,” she reminded everyone, “they assume they dialed wrong, not that the telephone network misdirected their call. That’s how reliable they think it is. They assume it will always work. They don’t understand the technology, but they assume we do. And we betrayed them. It was a betrayal. And an apology just won’t cut it.”
That’s why she pressed Bob Allen to personally brief the media, rather than delegating it to some unknown engineer who probably had a better understanding of the technical problem. That’s why she wanted the briefing at the company’s Network Operations Center. All those video screens were impressive and a great photo opportunity, but the real point was the white shirted managers in front of them. There was lots of technology here, but there were people too, and the people were in charge.
And that’s why she also pressed for something more—the company had inconvenienced its customers; it should give something back to make its apologies real. She knew it wouldn’t change anyone’s mind about AT&T all by itself but, combined with everything else, it would show that the company was at least trying to do the right thing. And that was one of her core beliefs—a belief handed down from the company’s first PR officer, Arthur W. Page—public relations is 90 percent doing, only ten percent talking about it.
The next day, at 11:00 a.m., Allen held a news conference at the Network Operations Center and explained what he knew about the outage at that point—it was a software error that the company should have caught. Bell Labs was working on a permanent fix. And to underscore how much AT&T valued its relationship with its customers, the company would ask the FCC for approval to declare February 14, Valentine’s Day, as an additional day of holiday calling, reducing normal rates by an average of 33 percent. It was a nice gesture, not entirely wasted on the cynical reporters, but they continued to pepper Allen with questions about the outage. His answer to one of those questions—totally unrehearsed—demonstrated what the man was made of. “Mr. Allen,” one of the reporters asked, “exactly who was responsible for this?”
Without missing a beat, Allen looked the reporter in the eye. “As far as our customers are concerned,” he said, “I’m responsible. And I’m going to make sure it never happens again.”
“After that,” Marilyn said. “AT&T employees were ready to carry him out of the room on their shoulders.” Most customers were also in a forgiving mood. As the publicity died down, the company measured its quality and reliability ratings among customers with some trepidation. They were unchanged, holding firm at their historically high levels. And Marilyn admitted to some sense of pride when the Wall Street Journal quoted her on how to handle a crisis: “Tell the truth, tell it fast, take accountability,” she said. “And give something back.”
The first three precepts were standard PR advice—when you have a problem, get ahead of it by making yourself the most reliable source of information about it. Otherwise, someone else controls the conversation and you’re always playing catch-up. But the fourth piece of advice was classic Marilyn. She knew that a company worthy of trust unlike a company that would settle for being perceived as trustworthy—would not be satisfied with simply making a heartfelt apology after letting its customers down. It would want to do something that demonstrated the sincerity of its regret.
Unfortunately, it did happen again, although not for the same reason and not on the same scale. On September 17, 1991, anticipating an unusually hot Indian Summer day, Con Edison asked AT&T to take one of its switching centers in lower Manhattan off the power grid. All the company’s switching centers had redundant power sources, including back-up generators and battery plants in addition to commercial power. To help the electric utility avoid brownouts, AT&T and Con Ed had long-standing agreements to move its switching centers to diesel-generated power on high-usage days. Instead of cutting over to a generator, however, the switching system began to draw power from a last-resort battery system.
Ironically, the switching center power crew, who would normally do a walk-through of the power plant when commercial power was switched off, went straight to an alarm training session when they reported to work that morning. Audible alarms, which should have sounded, had been disconnected when the power plant had been upgraded and never reconnected; a visual alarm bulb had burned out and had not been replaced.
Since the batteries were only designed to last six hours, by 4:40 p.m. they were essentially exhausted and the switching equipment began to shut down, disconnecting communications in and out of lower Manhattan and at the area’s three major airports, including between control towers and airplanes. It was 10:30 p.m. before all service was restored. And in accord with Murphy’s Law, the Chairman of the Federal Communications Commission was on one of the planes left circling the skies above New York when the control towers lost communications.
The next day, Bob Allen and Marilyn Laurie made an unpublicized visit to the central office and met with employees. The company’s government affairs people spent the day fielding calls from senior officials at the Department of Transportation, the Federal Aviation Administration and the Federal Communications Commission. The company’s media relations people could tell the media what happened, but how and why it happened were still a mystery. Somehow, triply redundant systems had failed. AT&T engineers descended on the switching center to figure out why.
Meanwhile, the executive responsible for sales to businesses felt strongly that his customers had been most directly affected. By late afternoon, he had talked his unit’s media relations team into holding a conference call with business media so he could reassure his customers by explaining what he was doing. The media had other ideas—they wanted to talk about all the people stranded for hours at airports and on airless, foodless planes. Pressed to explain how triply redundant systems could fail, the executive speculated that the switching center technicians had probably disabled the alarms because they were loud and had a tendency to trip even when there wasn’t a real problem.
The reaction was swift. As Marilyn later put it, “We got beaten up by the union and creamed by the press for blaming our employees.” By afternoon the next day, in a second media conference call, the executive actually responsible for network operations explained the processes that had failed and exonerated the employees. But the damage had been done.
Follow-on media coverage clarified the facts but repeated the sharp criticism for initially blaming low-level employees. The FCC demanded a written report within 10 days; the chairman of the House Commerce Committee announced he would hold hearings on the incident; New York City’s mayor demanded that major communications companies back each other up in the case of outages; and AT&T ultimately lost the FAA account.
The contrast between the two network outages was striking. Surveys showed that in both cases, after an initial drop in public confidence, AT&T eventually regained pre-outage levels of customer approval. But in the second instance, it took longer to recover, perhaps demonstrating that Marilyn’s approach was not a fluke. Bob Allen stood up and took accountability without pointing fingers at anyone else. The brash young business services president, by contrast, wanted to look “in charge” but he was too quick to point fingers.
It also demonstrated the limits of the management system Marilyn had adopted for her public relations organization. She had multiple internal clients, each with its own business goals, sometimes conflicting strategies, and varying levels of corporate loyalty. They all traded on the AT&T brand, but many of them put their personal brand first. Marilyn thought of the AT&T brand as a competitive moat separating the company from its competitors. It drove her nuts when business units drew water from the moat without replenishing it, through things like price cutting, or, as in this case, treating something like the power failure as a threat to their next quarter sales rather than a reputational issue for the whole company. She could not individually counsel all the business “presidents” created by the company’s decentralized line of business structure. And she definitely didn’t trust them to hire their own PR counsel.
So she had set up an organization of small PR teams dedicated to each business. The members of each team were co-located at the business’s headquarters, attended the business head’s staff meetings, and participated in the development of business strategy. They also attended Marilyn’s staff meetings and participated in the development of the corporate communications strategy. But she had advised the business unit PR teams to invest emotionally in their client’s success. “It’s a lot easier to be heard,” she said, “when people feel you care whether they win or lose.” The danger, as happened in the power system case, is that the business unit PR leaders identified so closely with the business unit that they lost a broader perspective, or they allowed themselves to be browbeaten into doing what their client wanted.
Sensing the pressure public relations people feel to please their internal clients, she cautioned her people not to let their interest in advocacy swamp their sense of what is open and honest. “We must not end up distorting information because we’re trying to make it look pretty” she said, “or because we’re trying to avoid a problem.” She offered two reasons. “First, it’s the right thing to do in serving the interest of our ultimate client—the public. And second, the truth always comes out. If you look like a liar or a dissembler in the process, no one will ever trust you again.”[2]
If that reasoning wasn’t enough, Marilyn had control over the salaries and promotions of all 500 people within her organization, whether they were on her payroll or a business unit’s. It was usually enough. Managing her colleagues in senior management, however, was a little more difficult.
September was usually the month when the AT&T board of directors held an off-site retreat, preceded the evening before by a dinner of all the company’s senior officers and their spouses. In 1991, it happened just after the New York power outage. “But the subject seemed to be off limits,” Marilyn recalled. “I finally leaned over during dinner and whispered to Bob Allen, ‘we have to talk about this.’”
After dinner, she cringed as Allen got up and said to the group: “Marilyn says we have to talk about the service failure.” The officers rather awkwardly left their spouses behind and trooped into another room. “Only then, as we faced each other in privacy, did the pain, anger, and shame of what we felt was happening come out,” she recalled. “The core of our identity was reliability. Our personal pride was built around flawless service. We were all feeling a terrible sense of loss . . . and we were in denial. We didn’t see what we didn’t want to see. And until we faced up to the unthinkable truth—that we had a systemic problem—we couldn’t act on the root cause.”
The public gave AT&T the benefit of the doubt twice because it had a hundred-year record of reliability. “That, to me, suggests the first pillar of trust—consistency, consistency, consistency,” Marilyn later said. “If you don’t do what you’re in business to do over the long pull . . . nothing else matters. And if you stumble, you’d better pick yourself up in a hurry and restore that consistency, because it’s the core of what people believe when they believe in you.”
Following that uncomfortable evening, the company’s senior team resolved to recapture the exceptional reliability standards for which the company had been known. “That was the triggering of the quality movement at AT&T,” Marilyn later said. “And in good old AT&T style, once we decide to do something, stand back because the whole ship is turning in that direction.” Indeed, the company hired dozens of quality professionals and began a massive retraining effort. Marilyn oversaw development of an internal communications program called “Ask Yourself,” which stressed individual accountability for protecting the company’s customers. “It was delivered in a dialog manner with lots of conversations led by senior officers to explain not only that quality was everybody’s job, but that there could never be enough rules to stop these things from happening if individual employees didn’t hold themselves accountable for the kind of service the old Bell System had stood for,” she remembered. “It was Angus MacDonald reborn in 1990.”
Marilyn had used the service crisis as an opportunity to refocus the company on one of its sources of strength. She could have been following the playbook of the canniest of politicians, Rahm Emanuel. “You never want a serious crisis to go to waste,” he told a conference of corporate executives just before his boss President Obama assumed office. “Things that we had postponed for too long, that were long-term, are now immediate and must be dealt with. This crisis provides the opportunity for us to do things that you could not do before.”[3] He was talking about crises that would confront the new president when he took office in a few weeks, e.g., recovering from the Great Recession, the cost of healthcare, etc. None of these crises were of Obama’s making but they would soon be his to deal with.
But there’s a corollary to Emanuel’s maxim: If you are prominent enough, you can be someone else’s lever. Marilyn learned that too. For example, in the late 1980s, when smart mobs were still using the U.S. Mail and door-to-door canvassing, one of the most prominent single-issue groups of all dragged AT&T into the most divisive national debate since the Vietnam War—abortion.
[1] REA is CEO Robert E. Allen; NOC is Network Operations Center.
[2] L. J. McFarland, L. E. Senn, and J. R. Childress, 21st Century Leadership: Dialogues With 100 Top Leaders, p. 126.
[3] Reported by Gerald F. Seib, “In Crisis, Opportunity for Obama,” Wall Street Journal, Nov. 21, 2008, https://on.wsj.com/3bkQJfT.
Marilyn: A Woman In Charge — Marilyn Laurie’s Life In Public Relations
By Dick Martin | Buy paperback | Buy hardcover | Buy Kindle edition | Buy audiobook
Marilyn: A Woman In Charge chronicles the ascent of Marilyn Laurie, a self-described “little Jewish girl from the Bronx” from a volunteer who helped start Earth Day to advisor in the corridors of power of a great American company. It illustrates the life of a woman in charge, through days of heady highs and humbling lows with grace and resilience.
