Detailed Information on the
Transportation Security Administration: Federal Air Marshal Service Assessment

Conduct an evaluation to look at the number of FAMS per flight, seating protocols, and the planned number of annual training and field office days by Q3 FY06.

Conduct an independent evaluation seeking to validate program structural elements and related performance targets by Q3 FY 2006. Specifically, the evaluation should assess and validate FAMS program performance related to flight coverage risk categories, the distribution of covered flights, and target levels of coverage.

Collect data on long-term and annual performance measures to allow for a reassessment of the program.

Measure: Percentage level in meeting FAM coverage target for each individual category of identified risk.

Explanation:Addresses general flight FAM coverage. Target performance is a uniform percentage level in meeting each individual coverage level for the risk categories (i.e, actual coverage reached xx% of coverage target).

Measure: Level of FAM days allocated to core mission (i.e., the number of days FAMS are flying on aircraft versus training and other activity days).

Explanation:This measure depicts the utilization rate of available FAM days for the core mission activity -- flight coverage -- as oppposed to training and field office days.

Measure: Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins with FAM coverage.

Explanation:

Measure: Level of operational FAMs verified as meeting recurrent training requirements.

Explanation:The program has determined that each FAM should receive 20 days of required training each year. The target therefore depicts 100 % of FAMs receiving the required level of training.

Measure: Level of operational FAMs who successfully complete Phase II training.

Explanation:The program has determined that each FAM should receive two layers of non-recurring, initial training. The purpose of the measure is to gauge management success in ensuring every current and new FAM receives both phases of training. Phase II is the second part of the 15 week basic training that all FAMs must take (Phase I is 8 weeks). Phase II is a 7 week agency specific course.

Measure: Level of FAM coverage for each identified category of risk.

Explanation:

Measure: Level of FAM coverage on flights with identified threats.

Explanation:

Is the program purpose clear?

Explanation: The purpose of the Federal Air Marshals Service (FAMS) is to enhance aviation security by providing a security presence during flight inside commercial passenger aircraft.

Evidence: Section 105 of the Aviation Transportation Security Act (ATSA) of 2001 specifically provides that TSA (1) may provide for the deployment of Federal air marshals on every passenger flight of air carriers in air transportation or intrastate air transportation; and (2) shall provide for the deployment of Federal air marshals on every such flight determined by the Secretary to present high security risks.

Does the program address a specific and existing problem, interest, or need?

Explanation: Specific and credible intelligence suggests that al Qaeda still actively seeks to conduct terrorist missions aimed at taking over U.S. commercial aircraft. At this point in time, it is not clear that other layers of security apart from air marshals are sufficiently robust as to adequately prevent a terrorist or team of terrorists from boarding an aircraft with capable weaponry. Should this occur, reinforced cockpit doors and air marshals provide a last line of defense for an aircraft.

Evidence: Evidence is classified.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: No other Federal, State, or local program provides a law enforcement presence on commercial aircraft.

Evidence: Section 105 ATSA specifically provides that TSA (1) may provide for the deployment of Federal air marshals on every passenger flight of air carriers in air transportation or intrastate air transportation; and (2) shall provide for the deployment of Federal air marshals on every such flight determined by the Secretary to present high security risks. No other law enforcement entity is authorized to provide on-board coverage of commercial air carrier flights.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: It is not clear the program is free of major design flaws. Key aspects of program design need independent assessment and validation. In particular, the FAMs program should validate its requirements on numbers of FAMS in a covered flight, the seating protocols, and the planned number of training and field office days.

Evidence:  

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The nature of the FAMS program is such that the entire flying public is intended to be the beneficiary of program resources. Therefore, this question is not relevant to the FAMS program.

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: A set of long term measures have been finalized addressing critical program areas including terrorist incident outcomes, flight coverage, operational tempo, and air marshal training.

Evidence: PART performance measure section.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long term targets are under development.

Evidence:  

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: A set of annual measures have been finalized addressing critical program areas including terrorist incident outcomes, flight coverage, operational tempo, and air marshal training.

Evidence: PART performance measure section.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Long term targets are under development.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Key partners supporting FAMS program goals include the FBI, terrorism task forces across the country, and other Federal law enforcement agencies. The FAMS program has established close working relationships with relevant organizations in each area.

Evidence: The FAMS program has an MOU with the FBI establishing the FAMS role as full participants in all of the Joint Terrorism Task Forces and the NJTTF located at FBI HQ. FAMS also participate with the Anti-Terrorism Task Forces sponsored by U.S. Attorney Offices around the country. FAMS coordinate with the Council of Governments and the National Capitol Region Coordinating Center for security activites related to aviation. The FAMS created and coordinate the Force Multiplier program to leverage other Federal law enfocement assets flying armed on commercial air carriers.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The FAMS program has not been in existance long enough to assess this question. To date, just one significant evaluation was performed, but the scope of that evaluation was narrow.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The FAMS program has not been in existance long enough to assess on this basis. The FAMS program has had just one budget submission to OMB and the Congress (FY 2004), but this was done in the early stages of the program's development. The FY 2005 budget cycle is the first 'normal' budget cycle for this program.

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The primary strategic planning deficiencies have been the lack of a strategic plan with adequate performance goals, measures, and targets. Meaningful actions have been taken to address these deficiencies.

Evidence: A draft strategic plan has been developed, as well as a related operational business plan. As part of the PART review, comprehensive performance goals, measures, and targets generally have been finalized.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The primary strategic management emphasis is flight coverage, including the identification of high risk flights to ensure adequate coverage, and maximizing air marshal days dedicated to core missions. Current data collection efforts meet management needs in these areas.

Evidence: The FAMS collects a range of pertinent performance information, such as monthly missions flown and aircraft incidents. In addition, FAMS mission operations liaison collect data from groups to include: the airport operators; Airline Pilots Association; Air Transport Association; executive offices of air carriers;and, other law enforcement agencies, regarding various interactions with FAMS personnel/operations.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: The TSA established a permanent performance management system that defines performance agreements for groups of employees at all levels, including TSA screeners, supervisors and executives. The Federal Air Marshal Service is a part of that system, and managers and partners will be held accountable for cost and performance results. The strategic planning process is refining specific long-term and annual performance targets which will be used to measure program and managerial effectiveness. Field office managers are required to provide headquarters with a work plan identifying annual program goals and fiscal requirements. Managers are evaluated based on their ability to accomplish the goals stated in the work plans.

Evidence: The TSA performance management system collects FAMS outcome and output data, field managers have specific performance goals included in annual workplans.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The FAMS program has not been in existence long enough to assess obligation data on this basis.

Evidence:  

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The FAMS program has procedures to ensure efficiencies and effectiveness. Air marshal scheduling is automated, and man hours are closely monitored. IT acquisition is managed centrally through a managed services contract in TSA. Administrative services are outsourced.

Evidence: A key efficiency peformance measure of the FAMS program is level of man hours allocated to core mission activities. The central management of information technology purchases of FAMS products by the TSA Office of the Chief Information Officer via a UNISYS contract ensures consistency, control, and a lack of duplication in services, equipment and expenditures. The FAMS Mission Scheduling System's automated SABRE system has replaced the time-consuming, expensive manual operation, making deployment more efficient and reducing the incidence of scheduling error. All travel vouchers, contracts, accounting system services and the SABRE program management are provided by the TSA Technical Center via an interagency service level agreement that delivers consistent, cost-effective service to the FAMS, as it makes unnecessary any duplication of those functions by the FAMS. Acquisition procedures require contract sourcing, and the procurement of cost quotes from at least three vendors prior to a purchase requisition.

Does the program collaborate and coordinate effectively with related programs?

Explanation: While the necessary collaboration for success in meeting FAMS goals is limited, the FAMS program does collaborate to a great extent with internal and external programs and activities that either have direct bearing on goal outcomes or will help ensure mission success.

Evidence: TSA assigned the FAMS responsibility for the operational management of the Federal Flight Deck Officer (FFDO) program. The FAMS provides 24/7 support and incident management to ensure full and effective coordination with the armed commercial pilots (FFDO) and the aviation industry. The FAMS created and coordinate the Force Multiplier program to leverage other Federal law enforcement assets flying armed on commercial air carriers. The FAMS manage TSA's Less-Than Lethal weapons program by responding to requests from air carriers to deploy LTL devices. FAMS participating in FBI-JTTFs; USAO-ATTFs; TSA CAPPS and screener working groups; and various executive Table-Top exercises. Daily FAM MOC communication with the FAA contributes to force efficiencies and critical incident management.

Does the program use strong financial management practices?

Explanation: The FAMS program appears to manage financial resources properly.

Evidence: No material weaknesses are attributable to the FAMS program.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Specific management-related deficiencies were identified in recent report of the Inspector General, and the FAMS program has begun taking meaningful steps in each area to address these problems.

Evidence: The FAMS response to the Inspector General report identified specific, responsive actions the organization had taken.

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation:  

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation:  

Evidence:  

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation:  

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation:  

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation:  

Evidence: