原创 Massive telecommunications breakdown

In broadcast engineering, one of the priniciples is that when something breaks you shouldn't focus on fixing it. Your first action is to get things back on-air! Also, you can't panic. Too many times I have seen talented technicians melt under the pressure of knowing that the systems they are working deliver TV to millions of people.

Anyone prone to moments of headless-chicken hysteria or paralysis through fear is simply not fit to work in this type of environment.

At the beginning of my career I worked 12-hour shifts running the satellite earth station master control room of a major European communications company. Engineers worked solo on an eight-day week with two day shifts, 24 hours off, two night shifts, and then three and a half days off. Frankly this was a great shift pattern, and I did loads of overtime during my off days.

On the day of this particular incident I was in the middle of a long run of overtime. I was pretty beat, but loving the extra cash! I handed things over to the day-shift engineer and went home to grab some sleep. At 1 p.m. my jangling phone jolted me out of a dream. "The Middle East has gone down," shouted the day shift engineer.

Now wide awake, I asked "What service?"

His reply cut through me like a knife: "All of them, and the MCR manager is on holiday. Can you come in?"

I threw some clothes on and literally ran halfway across central London to the office. He was right, almost every Middle-East service was offline. We had a number of large telcos in the Middle East and some call centres in South Asia that used us either as backup links or for premium rural customers. The day shift engineer was one of those guys who didn't do well under pressure, so I seized command and started digging in.

After a few hours of getting nowhere, I sent the day guy home because he was really just getting in the way. One of the senior managers had turned up because his phone was ringing off the hook. He was fielding calls from irate customers.

I stripped half the entire satellite earth station, bypasssing or reworking just about everything. This is very challenging when you can't actually see the other end of what you are transmitting to and have very little indication if what you are doing is even working.

I had to call people on sat-phones in Iraq who I knew would have spectrum analysers and ask them to help me. I had to call the satellite owners and ask for their help, but it took me hours. By 2 a.m. I had switched over half the systems. Quite frankly, I don't know or I forget what combination actually worked, but the system was back and customers were somewhat happy again.

The manager taking the calls went out to get me a kebab while I came down from my caffeine and adrenaline buzz. When he came back, he said, "Hey, isn't it your shift now?" Yes, at 8 p.m. the previous evening I was on shift, so I now had to do my shift until the next engineer came in at 8 a.m.! On two hours of sleep I was now finishing the final five hours of my very long day. Needless to say, that was a long night.

It wasn't the last time I did several days without sleep, but thankfully I never had another failure that big.

This article was submitted by Bob Hannent as part of Frankenstein's Fix, a design contest hosted by EE Times (US).