When a Salt Lake City router went offline, only government telecom contractor Harris knew that the backup card was not immediately available and one technician had access to where it was kept. Meanwhile, hundreds of aircraft and thousands of passengers were thrown off schedule as the lack of an FAA filing system left pilots submitting flight plans manually.
When the Federal Aviation Administration's national flight-plan filing system went down for 4 hours the morning of Nov. 19, disrupting the takeoffs of hundreds of commercial flights and throwing hundreds of thousands of travelers off schedule, it wasn't yet known that the culprit was one faulty part inside a telecom router link in Salt Lake City.
The faulty router, which for reasons not yet established was not able to default to a backup, also shut down a second major system node in Hampton, Ga., effectively bringing to a halt the inputting of flight plans filed by U.S. commercial pilots. Commercial aircraft cannot take off from a U.S. airport without filing a flight plan.
The glitch forced hundreds of pilots flying that day to enter their plans manually via e-mail or by faxing them into the system, causing widespread flight cancellations and delays.
Most flight plans are routine and pre-entered as a template in the system. Pilots normally make only few changes in their altitude, speed and directional plans, depending on weather conditions and the weight of the aircraft. When the templates are not available, the pilot is forced to reconstruct the entire flight plan, which is a tedious and time-consuming exercise.
When the router went offline, only the system maintainer—government telecommunications contractor Harris—knew that the backup card was not immediately available, and that one technician, who hadn't come to work yet that day, had the key to the storage closet where the part was kept.
So the FAA had to wait until this technician was able to come to the site in Salt Lake City to replace the faulty card inside the router, reconfigure the software, and get the communications backbone back up and running so that the nation's air traffic could get back to normal.
This information was supplied to eWEEK by the Professional Aviation Safety Specialists union, an affiliate of AFL-CIO that represents about 11,000 FAA technicians.
Does the failure of a single router that crimps a national telecommunications system sound ridiculous in this day and age of virtual links, automated processes and autonomic computing? It does. But that's what happened, and that's why the Department of Transportation is going to launch an investigation into this incident to see that this doesn't happen again.
Harris, the government contractor that installed and runs the FTI (Federal Telecommunications Infrastructure) system, is the entity responsible for the infrastructure connecting the nodes for the FAA's flight-plan system.
"If the FAA owned and maintained this system, the problem could have been corrected within minutes," PASS National President Tom Brantley wrote on the union's Website. "This could have reduced delays tremendously and allowed a much quicker resolution to the problem. Meanwhile, because it took so long for Harris to address the problem, delays continue to plague the system."
Before 2002, when the FAA contracted out the FTI system to Harris, the system was maintained by FAA telecom technicians on duty 24 hours per day.
"[Before 2002] the only thing that the FAA used to contract out was the line services, belonging to MCI, Verizon or whichever company was the local supplier," Chuck Siragusa, a PASS spokesperson, told eWEEK. "The on-site FAA technicians are well-trained in mission-critical systems, routers, modems, all of it. If this [FTI] system had been maintained by the FAA, the impact [of the Nov. 19 outage] would have been minimal, because a fix could have been made much quicker."
The FAA recently spent millions of dollars updating its antiquated Philips mainframe system with a new one that uses Stratus Technologies high-performance servers and other elements from Sun Microsystems, Cisco Systems and other first-tier IT suppliers. The old system, which went online in 1988 and served the FAA well for two decades, was approaching its end of life and had suffered a series of breakdowns in the last few years. However, the new IT system was not the issue Nov. 19.
The FAA utilizes the NADIN (National Airspace Data Interchange Network) communications link for the flight-plan system. The two NADIN sites in Salt Lake City and Hampton, Ga.—along with including the 21 other FAA IT stations—no longer use a multipath communications backbone composed of many different redundant links.
As mandated by the Bush administration in 2001, all the communications links that previously were government-owned and maintained by FAA employees were contracted to Harris, under the $2.4 billion FTI contract.
Rep. Jerry Costello issued the following statement Nov. 19 regarding the outage: "While today's incident could have been much worse, anytime you have a system-wide outage it needs to be thoroughly reviewed and it brings up several questions that the FAA needs to address. Why did it take four hours to locate a seemingly small technical problem, and why did it have a system-wide effect? Is the FAA's oversight of its contract with the Harris Corporation sufficient? The relationship between the FAA and its vendors is a critical one, given that the transition to the Next Generation Air Transportation System will require more such partnerships. Our staff is discussing these questions with the FAA and we will continue to explore these issues. In addition, Chairman Oberstar and I have asked the Department of Transportation Inspector General to conduct a 60-day study of the outage and FAA's corrective action plan."
Harris spokesperson Marc Raimondi told eWEEK that people should keep in mind that weather conditions cause most flight delays, and that the FTI system used by the FAA has a very good performance record. "Five nines—maybe even nine nines of efficiency," Raimondi said.
Raimondi issued the following statement from Harris: "We're working with the FAA to evaluate the interruption in order to prevent similar outages in the future. FTI has proven to be one of the most reliable and secure communications networks operating within the civilian government. Safety and security are the highest priorities."
Aug. 15, 2009
See the makeup of the floor plan at ORD pages 1,
2, 3,
4 for workstations
and the FAA
order on space
permitted/allowed/mandated per tech.
Since July 2003:
For the OTHER PASS Bargaining units- we've been at Impasse since July of 2003:
IMPASSE ISSUES BETWEEN PASS's 4 bargaining units AND FAA
The Professional Airways Systems Specialists and the Federal Aviation
Administration are at impasse in their contract negotiations in the following
bargaining units – Flight Standards (“FS”), Aviation Standards (“AVN”),
AFS-700 and Manufacturing Inspection (“MIDO”). Employees in these bargaining
units are employed across the United States in various FAA facilities. Many of
the issues over which the Parties are at impasse are already included in other
collective bargaining agreements in the FAA, including the agreement between
PASS and the FAA covering the Airway Facilities bargaining unit. The FAA
recently stated that it was preparing to submit the impasse to Congress so that
the FAA would be in a position to impose the contract on FAA employees without
further bargaining. The following is a list of issues over which the Parties are
at impasse.
1. Child Care Subsidy – The Parties have been unable to agree on the
issue of adjustments in family income level necessary to qualify for child care
subsidies.
2. Flexible Spending Accounts – The Parties have been unable to reach
agreement on extending this program to PASS’s bargaining units. During the
last bargaining session, the FAA refused to discuss the issue.
3. Hours of Work – Bargaining unit employees have raised issues relating
to the impact on their personal lives of changes in their normal work schedules.
The Parties have been unable to reach agreement on PASS’s proposal to solicit
qualified volunteers for these assignments to lessen the impact of such changes.
The FAA now takes the position that it has no duty to bargain over this issue.
4. On-The-Job Training – The Parties have been unable to reach
agreement on a comprehensive contract article concerning the selection of OJT
instructors, the delivery of OJT and premium pay for OJT instructors similar to
programs agreed to by the FAA in other bargaining units.
5. Professional Liability Insurance – Congress authorized agencies to
reimburse certain inspectors for part of the cost of personal liability
insurance to cover legal costs not assumed by the federal government. The
Parties have been unable to reach agreement on the implementation of this
program for its inspectors in the Flight Standards and MIDO units.
6. Reimbursement for Educational Expenses – The Parties have been
unable to reach agreement on a comprehensive article dealing with reimbursement
for educational expenses. There is a disagreement over the fundamental goals of
the reimbursement program. PASS believes the program should focus more on career
development and education, rather than existing job training.
7. Sick Leave Buyout – PASS’s proposal would permit employees who are
participants in the Federal Employees Retirement System (FERS) to receive a
buyout upon retirement of 40% of the value of his/her accumulated sick leave.
The FAA has agreed to this for employees in other FAA bargaining units. The FAA
refuses to extend this arrangement to employees in PASS’s bargaining units.
8. Temporary (Virtual) Assignment to Virtual Certificate Management Office (CMOs)
–Several bargaining unit employees in Flight Standards work in a “virtual”
environment. As such, many matters applicable in a traditional work environment
do not apply or must be modified for a virtual work environment. Although the
Parties have reached understandings about several matters on this subject, the
Parties have been unable to reach agreement on the integration of these
employees into their “host offices,” assisting employees in their efforts to
become competitive and qualified for virtual assignments, and adjustments to the
Parties’ grievance procedure to accommodate the unique aspects of the virtual
environment.
9. Existing Memorandums of Agreement – In the FS unit, the Parties have
agreed that all existing memorandums of agreement/understanding will be void
upon implementation of the new master collective bargaining agreement, except
for those specifically attached as an appendix to the new agreement. The Parties
have been unable to agree on the list of memorandum agreements to attachment to
the new agreement.
10. Duration of the Agreement – The Parties have been unable to agree
on the duration of the new collective bargaining agreements. The FAA now seems
to be indicating that it will include this issue in its submission to Congress
so that it can impose the contracts on its employees for whatever duration is
wishes.
11. Implementing Memorandum of Agreement – The new agreements will
include major sections dealing with converting employees to new pay systems,
including pay-for-performance features, and new performance management systems.
In PASS’s negotiations with the FAA in the Airway Facilities bargaining unit,
certain details of the new agreement were included in a separate memorandum
agreement referenced as the Implementing MOA. The Implementing MOA would, among
other things, establish joint workgroups dealing with the details of the new
systems. PASS proposed a similar Implementing MOA for the four open agreements.
Despite bargaining over and agreeing to these subjects in other contracts, the
FAA now refuses to bargain with PASS over the following issues, which would be
part of the Implementing MOAs:
* Staffing Agreement – Insures safe and efficient staffing levels,
particularly in the inspector workforce.
* Employment Guarantee – Protects existing employees against layoffs during
term of contracts.
* Alternate Dispute Resolution (“ADR”) procedure for employees who are
converted to the new pay system – ADR process designed to save money and
provide quick and efficient resolution of employee complaints relating to
conversion to new pay systems.
* Superior Contribution Increase (“SCI”) system – Pay-for- performance
component of proposed new pay system. FAA wishes to impose standards without
bargaining with PASS.
* Treatment of Air Traffic Revitalization Act (“ATRA”) pay – Insures that
ATRA pay will be factored into pay for those currently receiving ATRA.
* Pay adjustments during the life of the CBA – The FAA refuses to bargain over
pay adjustments similar to those agreed upon in other FAA collective bargaining
agreements.
12. Pay System – Under FAA Personnel Reform, the unions representing
FAA employees were given the authority to negotiate new pay systems covering its
bargaining unit employees. It is important to obtain “employee buy-in” for
the new system. However, the FAA wishes to impose the Core Pay System on
PASS’s bargaining units despite major flaws in the system, which is strongly
dislike by FAA employees covered by the Core Plan, including managers and
supervisors. The FAA refuses to discuss any modifications to the Core Plan, even
though it has agreed to such modifications in other bargaining units. The
Parties have been unable to agree on several structural components of new pay
systems for these bargaining units. For example, some of the pay system issues
in dispute are as follows:
* Incorporation of Air Traffic Revitalization Act (“ATRA”) pay into base
pay.
* Minimum Annual Increases. PASS proposed that bargaining unit employees would
receive annual pay adjustments at least equal to the President’s Annual
Comparability Increase, to be incorporated into the Organizational Success
Increase (“OSI”). The FAA proposes to provide a discretionary annual
increase “up to” the amount of the presidential increase based on the
achievement of performance goals outside the control of bargaining unit
employees. The FAA’s proposed system is not “performance based” because
annual pay increases are not linked in a real way to the performance of the
impacted bargaining unit employees.
* Superior Contribution Increases (“SCI”). PASS favors an SCI system similar
to that which was negotiated in its FAA - Airway Facilities bargaining unit, and
which has been briefed to the FAA Administrator as a true pay for performance
system. Under this system, all bargaining unit employees who satisfy defined
individual performance criteria are eligible to receive an annual SCI pay
increase from a pool of money formerly paid as part of step increases in the GS
pay system. The FAA proposes a tiered, quota system that artificially limits the
number of employees eligible to receive an SCI increase, regardless of the
quality of their performance. Final SCI criteria cannot be negotiated until the
Parties have agreed on Career Level Definitions, which the FAA refuses to
discuss with PASS.
* Promotions. PASS proposed that an employee receive an 8% pay increase when
promoted from one level to the next, and a 16% pay increase when promoted two
levels. A two level promotion would be comparable to being promoted from a GS-9
position to a GS-11 position under the GS pay system. Under the promotion
section of the previously negotiated pay system covering the FAA-Airway
Facilities bargaining unit, the FAA has misconstrued the Parties’ intent and
limited the pay increase of an employee who received a two level promotion to an
8% increase. The FAA refuses to rectify this obviously inequitable practice and
the Parties remain far apart on this issue.
* Locality Pay. PASS proposed maintaining the current “locality pay” system,
and the FAA proposed deleting locality pay under the new pay systems.
* On The Job Training (“OJT”) Instructor Premium Pay – PASS proposed that
OJT instructors receive premium pay in the amount of 10%, as provided in other
collective bargaining agreements between PASS and other unions in the FAA. The
FAA refused to agree to OJT premium pay.
* Flight Premium Pay – PASS proposed that bargaining unit employees who are
members of FAA flight crews or who fly in an aircraft for the purpose of meeting
Event Based Currency (EBC) requirements, administering or receiving proficiency
checks, administering or receiving flight training, conducting aircraft flight
testing, or for any other purpose which involves serving in the capacity of a
pilot receive flight premium pay in the amount of 10%. The FAA refused to agree
to flight premium pay.
* Sunday, Night and Training Pay. In the MIDO unit, the Parties have been unable
to agree on these components of the pay system.
13. Career Level Definitions – Positions in the FAA will no longer be
defined by traditional position descriptions. Rather, the various levels of each
job category occupied by bargaining unit employees are defined by Career Level
Definitions (“CLD”). Although there is no dispute between PASS and the FAA
relating to CLDs for most of the bargaining unit positions, there are
substantial disagreements over the CLDs for the “specialized” job category
covering the position – Aviation Safety Inspector, particularly relating to
levels 3 and 4 of that category. PASS’s primary concern is that the CLDs
proposed by the FAA do not accurately reflect the duties and responsibilities of
inspectors who would occupy level 3 and 4 positions. As a result, promotions
would be artificially depressed and career enhancing opportunities will be lost
for many bargaining unit employees.
14. Pay Levels – The Parties previously reached an agreement concerning
periodic pay increases for this bargaining unit, but the FAA revoked its
agreement. In conformance with pay increases negotiated in the AF bargaining
unit, the agreement provided for annual 1.5% pay increases for bargaining unit
employees in the specialized job category over and above other pay increases set
forth in the pay system.
15. Performance Management System – Despite repeated requests, the
Agency has failed and refused and continues to fail and refuse to provide PASS
with a copy of the performance management system covering these bargaining
units. This has interfered with bargaining because the performance management
system is related to career level definitions and other aspects of the personnel
system.
16. Awards Pool – In the MIDO unit, even though the Parties have
tentatively agreed on some aspects of the article on awards, the Parties have
been unable to agree on the establishment of a pool of funds for awards, and a
section making it clear that awards are separate from pay-for-performance
increases under the PASS-MIDO pay plan, over which the Parties are also at
impasse.
17. Effective Date of CBA and Implementation Date – The Parties have
been unable to reach agreement on these dates.
1/20/09
Article 41 Grievance
I just received word from the arbitrator that he needs until 2-15-09 to issue
his decision in the Article 41 national grievance. I stressed to him that this
is a continuing issue and that union officials and members are getting
impatient. In the meantime, anyone who has been forced off from work on their in
lieu of days should continue to file local grievances so that we will have a
good data base for back pay if the arbitrator sustains the national
grievance.
Mike Derby