Monday, September 29, 2008

Things Gone Awry

ATC is a complicated, dangerous, challenging job... and that's when everything is working correctly. There are so many technical things in the balance - radars, communications, data lines, navigation aids, airport lighting, etc. - that it's fairly common to see things fail or malfunction. When that happens, you're required to work with it and continue moving your traffic safely.

Usually the problems - and their resulting workarounds - are simple. If one of your runways is closed, you use another one. A dying radio involves you moving pilots to a different frequency. A navigation aid failure may just result in you substituting a heading in lieu of "Proceed direct [NavAid]". These are the kind of things you'll see up on your information board above each radar scope, in our Outages and/or Remarks sections. They don't make your life especially difficult, but they do alter your operation somewhat.

However, what happens when things really go wrong? How do you react, change your tactics, and keep moving traffic? I found that out first-hand on Friday. Let's just say it was an interesting day....

Early morning.

I arrive at work around 9:15am. Things are working more or less normally. It's a beautiful VFR day and the traffic is typical Friday: steady but not as plentiful as the rest of the week. We have all of the following equipment available to us.


Maintenance pulls all of the main transmitters. We're now working on the secondary backup radios, except for two frequencies that are completely out of service (OTS) so MX can swap the parts. The operation is not really affected; airplanes using those two OTS freqs are just moved to different frequencies.

Now our equipment looks like this:

2pm. Commence poo flinging at nearest fan.

The day is moving along. All of a sudden, commotion breaks out in the radar room. Maintenance folks storm in and there is lots of discussion between them, the controllers, and the supervisors. Somewhere, somehow, we've had a data cut. The T1 data line that carries the data in and out of our building is completely dead. That's very, very bad news.

We've lost all data links with external facilities. The National Airspace System is essentially a huge computer network, and we've lost our connection to it. We have zero computer interaction with Jacksonville Center, Houston Center, Eglin Approach, or Mobile Approach. Our computers also aren't communicating with the control towers within our airspace, namely Pensacola Regional, Whiting NAS, and Sherman NAS.

What does this mean in concrete terms?
  • No flight plans are being transmitted to us from the Jacksonville Center host computer or any of the other facilities within or outside of our airspace.
  • Due to the above, no flight progress strips are printing out on our end.
  • We cannot take or receive handoffs via our radar scopes.
  • We cannot make amendments to existing flight plans via our Flight Data Input/Output (FDIO) systems.
  • Our local database is no longer being updated, so there are no updates to our arrival/departure tab lists on our scopes.
Remember what the equipment image looked like a few hours ago? This is what we have left:

Our radar scopes themselves, our backup radios, and our land lines. That's it.

The Big Picture

Inbounds and Overflights: We have aircraft coming towards our airspace from throughout the southeastern United States. All kinds: airliners, corporate jets, general aviation, military. Without that data line, we get no printed strips and therefore no information on any of these planes. We have no idea who they are, what code they're squawking, where they're landing, or what they're doing. We are in the blind.

Outbounds: At the same time, we have departures wanting to come off our airports for which we have no strips. Our airports' FDIO systems are working fine, so the flight plans are in the National Airspace System. However, they're not transmitting to us due to the data cut. As a TRACON, we obviously have to talk to these departing aircraft, but we have no clue what they're supposed to squawk, where they're going, or even what their call sign is.

General Automation: Everything that is supposed to operate via computerized automation is dead. No flight plans, no hand offs, no amendments, nothing. If we need to amend someone's altitude or routing, we have to call someone else to do it for us.

The Workarounds

Before we can continue working traffic, our functioning equipment is assessed.
  • We still have our radar, our (secondary) radios, our scopes, and our land lines, so we're not exactly ATC-Zero. We can still talk to other facilities without a problem.
  • We also have full internal functionality, so if we tag an airplane manually we can still hand it off or point it out to other sectors within our facility.
  • We can also still generate local squawk codes for aircraft that won't be leaving our facility's airspace.
Workarounds are therefore put in place to deal with the situation.

Manual Coordination
  1. Without automation, we have to perform all handoffs manually - regardless whether they're outbound or inbound - via the inter facility voice lines. It was definitely good practice for that phraseology: "Eglin North, Pensacola Whiting, manual IFR handoff. Ten miles southwest of Crestview VORTAC, squawking 4255, call sign VV1E097."
  2. Whenever an aircraft departs landing somewhere outside our airspace, we have to tell the departing tower to send a Departure Message via their FDIO to let other facilities down the line know that the aircraft in question is airborne. That results in strips printing out at the receiving facilities.
Cut Controller Workload
  1. Add a handoff controller to our busier radar positions, namely the two Pensacola scopes and the Sherman NAS scope, so that the radar controller doesn't have to do all the manual coordination himself.
  2. Immediately stop automatic releases for IFR departures from airports within our vicinity. That gives us complete control over when an aircraft will depart, so if we're going crazy coordinating we won't have a surprise IFR airplane pop up.
  3. Cease all practice approaches within our airspace, forcing all of our Navy trainers to do full stops. We frankly have more important things to worry about than whether or not Mr. Student Pilot gets his lessons done today. I don't mean that in any offense to our Navy or civilian trainees out there, but it's simply a matter of priority and safety.
Voice Transmission of Flight Plans
  1. When an airport is activating a departing aircraft, we have them call us with just the basic information:
    1. Call sign
    2. Type
    3. Departing Airport
    4. IFR/VFR
    5. Requested Altitude
    6. Destination
    7. First Fix outside of our airspace (for our routing purposes)
    8. Beacon Code
  2. The Centers, Eglin, and Mobile call us over their respective shout lines to convey arrival information to our Flight Data person. "Pensacola Data, Eglin South, Flight plan."
  3. Our Flight Data person hand-writes strips for each departure or arrival by hand, and then deposits then next to each radar position.
  4. Flight Data also manually enters the call sign, beacon code, and departure gate for each plane at each scope. While our Tab List is no longer fed from the NAS, we can type it in by hand so departing aircraft will tag up locally.

    For instance, if it was a Navy bird departing South Whiting, going out the Sikes gate, and departing from the "Z" position, FD would just type the following on any radar scope's keyboard to put the aircraft in the appropriate tab list:

    VV7E113 (call sign)
    ΔSIK (delta symbol + 3 letter departure gate abbreviation)
    1234 (beacon code)
    B06 (aircraft type)
    Z (departing sector in whose tab list it should appear)
And so it goes....

It was quite an experience. Having never seen anything like this before, it was pretty cool how everyone pulled together and just worked around the problems. Sure, no one was thrilled, but things got done. There wasn't much anyone could do about it other than get through the situation. At times it got pretty tense, but everyone adjusted to the new operation.

For the pilots, all of these operational workarounds were completely invisible. Whether it was a Delta wanting to land at Pensacola Regional, a Cessna 182 en route to New Orleans, or a Navy helicopter departing on a cross country to Panama City, they didn't suspect a thing. Nor did they have a reason to. All they heard was the usual phraseology, despite the controllers having to do everything by hand. Thanks to the efforts of the controllers here and our neighboring facilities, the operation kept going and the traffic kept flowing.

The traffic started to die down a bit towards the end of the day and those of us who had stayed for overtime got to go home. Last I heard, it was going to take up to a couple days to get the data line back in operation. Thankfully, the weekends over here are relatively slow since the Navy doesn't fly too much those days. That should make it somewhat easier on the folks working this Saturday and Sunday.

As they say about ATC, every time you plug in you've got to be prepared for anything, and that means anything on either end of the radio. You can't just throw up your hands and walk away. People are depending on you to get the job done with whatever means you have at your disposal.


Anonymous said...

Is there any news coverage of this? Seems like a big deal.

Wicked Penguin said...

You wouldn't be hearing about this on the news. We didn't lose complete service and, like I said, it looked normal to the pilots from the outside.

Let's compare it to, say, the Memphis Center failure last year. ZME's problem was a telephone line cut. All of our land lines to other facilities are essentially telephone lines, so they lost voice communication with other facilities. In addition, they lost connection with three of their radar sites along with their data connections.

That's a huge safety issue, since A) they can't see some of their planes, B) they can't coordinate via their scopes, and C) they can't coordinate verbally. The building was completely cut off from the outside world. Therefore, they went ATC-Zero.

We, on the other hand, still had our voice lines and still had both our radars. We could see all the planes, and with flight plan data manually passed to us over voice lines we could figure out who they were. And when it came time to hand them off to another facility, we just did it via the shout lines.

Therefore, unlike Memphis, we didn't really have all that great of a safety risk. We just needed to work harder and implement unusual procedures to make it happen.

Ironically, from what I was told, what we were running on Friday is how ATC was run before all of the automation was developed. So, I guess you could just say we went "old skool". :)

Anonymous said...

The people that keep the syatem always have and will continue to do whatever it takes to get it done. That has always been our strength. That is why it is soo disapointing that the leaders of the agency have (and continue) hurt the workers. Hopefully one day their actions will come back to them just as wall street looks for a bail out so will the leaders of our agency.

Anonymous said...

You are such a friggin goober. I worked Eglin with nothing...including radar! You probably have no idea what the hell non-radar means, do ya? You're the guy that sits next to me and just hands me his strips when you're friggin screen goes blank.
will you go to Sammy's and get a life? Please? You embarass me.

Anonymous said...

I always said, Technical Operations, formerly known as Airway Facilities need to be contracted out as soon as possible to avoid any reoccurence of this scenario. We(ATC) could use that money being "wasted" on Technical Operations employees and use it for our raises once Jane Garvey gets back in. I have two BMW's and a 55 ft'er thanks to Ms. Garvey. I will do everything and anything she asks for a bigger raise next go around for all of "our" troops, including the "new" hires.


Dan said...

Anonymous said:
"I always said, Technical Operations, formerly known as Airway Facilities need to be contracted out as soon as possible to avoid any reoccurence of this scenario."

Please...contract out Tech Ops...what are you on? Who do you think is responsible for having all that redundancy in place...the reason why a T-1 outage didn't put you at ATC-Zero? You don't have any idea how many systems have to be operational so you can do anything at all while sitting in your air conditioned room, crying because your too hot or too cold. What a joke. The fact is you're way over-paid. The money wasted on your sorry ass would have been better spent on a decent contract for telco as opposed to FTI Harris, who was at fault here not Tech Ops.

In regards to the original post: nice write-up!

Anonymous said...

Controllers are one-trick ponies giving 5 cent rides. If I were to get contracted out, I would at least have a marketable skill to do well elsewhere. You on the other hand, could maybe be a traffic cop, school crossing guard, or a trucking company dispatcher. The contracting-out sword cuts both ways, just ask some of your former FSS folks.

Anonymous said...

Unfortunately, I, like you, are entitled to our "own" opinions. In my opinion, after working nearly 20 years in different LARGE facilities, it is a STRONG belief that Technical Operations, formerly known as Airway Facilities, could probably be "contracted" out, yes, "CONTRACTED" out where the FAA would receive a BIGGER bang for its BUCK. How many times have I seen Technical Operations UNABLE to resolve "simple" problems where "EXPERTS" from Atlantic City, Oklahoma City, Regional personnel, or at worst, CONTRACTORS who designed these systems, like ASR 9/11, ASDE, and TDWR...Oh, I forgot STARS are called in to "fix" this equipment when there are problems! From my "viewpoint" as a Taxpayer, I truly believe that the Federal Government could save HUNDREDS of millions of dollars, some of which can be used to fund NEXTGEN and part can conceiveably be used to get us, Air Traffic, a FAIR contract where OUR skills are RESPECTED!!!!
I did not want to get into "mud slinging", however, I feel a responsibility to address some comments that were getting "kinda" personal...... It best can be described through a simple example.

We have a "person"(I want to be "politically correct"), whose job apparently in Technical Operations is walking around with a Starbucks Coffee in "their" hand(again I wish to be "politically correct") once a month and "signs" off some sort of paper on the fire extingusihgers. This same person other responsibility is to maintain some sort of Engine Generator/Back Up Power system in case of commercial power failures(i.e. Thunderstorms), but everytime it is called upon, the system doesn't work, and WE, Air Traffic Control end up switching positions, consolidating positions, and reducing our seperation because the equipment that we use is NO longer working because this "BACK UP" power system always fails.

Finally, the companies(2) that made both systems came and determined that the Technical Operations folks haven't been doing their maintenance properly and one of the gentlemen stated that his company could maintain and guaranteee 99.9% relaibility with a restoration time "GUARANTEED" within 60 minutes during the other 0.1% of the time.

All I am trying to say, is that I hope that Jane Garvey is our next TRUE "HEAD" BOSS where she can "direct" the FAA Administrator the way WE direct and control traffic.

Remember us when you fly to see Grandma and GrandPapa this Holiday season!


Anonymous said...

As far as contracting out goes, technology is not very far from removing AT's butts from their warm seats. There is not a decision that they make that cannot be programmed into a computer and later regurgitated to aircraft. These new computer employees won't complain about fatigue when working 4 out of 8 hours either. Nexgen is the beginning of the end for you guys. There used to be traffic cops too before us "technical" types figured out how to replace them with lights. Oh, and we may have sorry employees doing easy tasks like fire extinguishers, but you guys are guilty of the same. You just hide your rejects by moving them around from one small tower to the next.

Anonymous said...

I've said it for years.

One day, I'll replace the controllers with a switch.

That time is getting closer and closer.

Anonymous said...

Ohhhhhhh, Poooooooorrrr, Babyyyyyyyy! Did we hurt your feelings??? Get over it! You thin skinned weenie!