Detailed Information on the
Adolescent Family Life Program Assessment

Development of core data instruments and implement for 2006 grantee reporting.

Develop performance baselines, measures, and targets based upon data collected from core instruments.

Review the similarities between the AFL and the ACF Community Based Abstinence Education programs and recommend changes to reduce the redundancy of multiple funding sources for similar purposes.

Obtain approval for AFL efficiency measures.

Obtain approval of end of the year reporting templates for grantees that incorporate reporting of AFL performance measures/efficiency measure.

Develop a standardized set of criteria and specific protocols for review of independent program evaluations.

Provide targeted and constructive feedback to AFL grantees regarding their 2007 end of year program and evaluation reports.

Measure: Increase the involvement of parents in the lives of their adolescent children measured by the change in the proportion of AFL Prevention demonstration project clients who communicate with their parents about puberty, pregnancy, abstinence, alcohol, and/or drugs.

Explanation:Parental involvement will be tracked through the core data instruments by comparing baseline to follow-up data each year. The Core Data Instruments were distributed to grantees in October 2005 to uniformly gather FY 2006 baseline and follow-up aggregate data. The End of the Year Report template was approved by OMB in the spring of 2006. All AFL grantees are required to use this template to report to the OAPP by December 2006. This template will gather aggregate data from the Core Instruments being implemented for all new projects funded in FY 2004 and later. Targets will be set by Spring 2007 after end of the year reports are assessed for FY 2006. Measured by the proportion of AFL Prevention demonstration project participants who report communication with their parents about sexuality and other health and social benefits.

Measure: Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity as measured by the change in the proportion of AFL Prevention demonstration project clients who indicate that it is important to them to remain abstinent until marriage.

Explanation:The percent increase will be measured through the core data instruments by comparing baseline to follow-up data each year. The Core Data Instruments were distributed to grantees in October 2005 to uniformly gather FY 2006 baseline and follow-up aggregate data. The End of the Year Report template was approved by OMB in the spring of 2006. All AFL grantees are required to use this template to report to the OAPP by December 2006. This template will gather aggregate data from the Core Instruments being implemented for all new projects funded in FY 2004 and later. Targets will be set by Spring 2007 after end of the year reports are assessed for FY 2006. Measured by the proportion of AFL Prevention demonstration project participants who report communication with their parents about sexuality and other health and social benefits.

Measure: Reduce the incidence of repeat pregnancies among clients in AFL Care demonstration projects as measured by the proportion of project clients with a repeat pregnancy at annual follow-up.

Explanation:The proportion of clients who have a repeat pregnancy will be tracked annually and compared to previous years as well as trends in national data. The Core Data Instruments were distributed to grantees in October 2005 to uniformly gather FY 2006 baseline and follow-up aggregate data. The End of the Year Report template was approved by OMB in the spring of 2006. All AFL grantees are required to use this template to report to the OAPP by December 2006. This template will gather aggregate data from the Core Instruments being implemented for all new projects funded in FY 2004 and later. Data from collected pre-tests will be reported by December 2006. Annual follow-up data will not be available until the following year. Most care projects began implementing the core instrument sometime after January 2006 with the baseline survey. Clients will not be eligible for annual follow-up until 12 months after program entry (at least January 2007). As a result, targets will be set by Spring 2008 after end of the year reports are assessed for FY 2007. Measured by the proportion of clients reporting a repeat pregnancy at annual follow-up.

Measure: Increase AFL Care demonstration project client conformance with recommended infant immunization schedules as measured by the proportion of project clients whose infant has received all recommended immunizations at annual follow-up.

Explanation:The proportion of clients who conform with recommended infant immunization schedules will be tracked annually and compared to previous years as well as trends in national data. The Core Data Instruments were distributed to grantees in October 2005 to uniformly gather FY 2006 baseline and follow-up aggregate data. The End of the Year Report template was approved by OMB in the spring of 2006. All AFL grantees are required to use this template to report to the OAPP by December 2006. This template will gather aggregate data from the Core Instruments being implemented for all new projects funded in FY 2004 and later. Data from collected pre-tests will be reported by December 2006. Annual follow-up data will not be available until the following year. Most care projects began implementing the core instrument sometime after January 2006 with the baseline survey. Clients will not be eligible for annual follow-up until 12 months after program entry (at least January 2007). As a result, targets will be set by Spring 2008 after end of the year reports are assessed for FY 2007. Measured by proportion of project clients who report their infant has received all recommended immunizations at annual follow-up.

Measure: Increase the educational attainment of AFL Care demonstration project clients as measured by the proportion who have enrolled in or completed a high school or GED program at annual follow-up.

Explanation:The proportion of clients who continue their educational attainment will be tracked annually and compared to previous years as well as trends in national data. The Core Data Instruments were distributed to grantees in October 2005 to uniformly gather FY 2006 baseline and follow-up aggregate data. The End of the Year Report template was approved by OMB in the spring of 2006. All AFL grantees are required to use this template to report to the OAPP by December 2006. This template will gather aggregate data from the Core Instruments being implemented for all new projects funded in FY 2004 and later. Data from collected pre-tests will be reported by December 2006. Annual follow-up data will not be available until the following year. Most care projects began implementing the core instrument sometime after January 2006 with the baseline survey. Clients will not be eligible for annual follow-up until 12 months after program entry (at least January 2007). As a result, targets will be set by Spring 2008 after end of the year reports are assessed for FY 2007. Measured by the proportion who report enrollment in, or completion of, a high school or GED program at annual follow-up.

Measure: Improve the quality of the independent evaluations, required by statute, of Title XX prevention and care demonstration projects as measured annually by an independent review of grantee end of year evaluation reports.

Explanation:The proportion of annual evaluation reports that meet a standardized set of criteria (i.e., clearly stated hypothesis; adequate sample size; viable comparison strategy; appropriate statistical analyses), as determined by a team of external evaluation consultants, will be tracked annually and compared to past years. Targets will be set by Spring 2007 after end of the year reports are assessed for FY 2006. Measured by proportion of annual evaluation reports that meet the standardized set of criteria.

Measure: Sustain the cost to encounter ratio in Title XX prevention and care demonstration projects.

Explanation:Measurement is ratio of Costs/ Number of Encounters. Costs = expenditure of Title XX project funds for service provision. Number of Encounters = number of service hours delivered. The measurement will be separate for Prevention and Care demonstration projects. Because Title XX is a demonstration program, funded projects use a wide range of approaches and vary greatly in the intensity of service delivered. Not only does intensity of service vary from one demonstration project to another, it can also vary within a project depending on the differing risk profiles among clients. Using encounters for this measure, rather than just number of clients, helps correct for this. This reduces the chance of the unintended consequence of keeping cost down by limiting services and reducing quality. The cost of implementing Title XX demonstration projects increases annually due to inflation and other factors. For example, both care and prevention demonstrations tend to be personnel/staff intensive and must keep salaries at a level to recruit and retain competent staff. Care projects serve pregnant and parenting adolescents and their infants; required core services include prenatal, postnatal and pediatric care, as well as educational and social services. Medical services, in particular, add substantially to the cost of care demonstrations when compared to prevention demonstrations which tend to focus on education. Prevention demonstrations have evolved from being largely curriculum-driven to also incorporating added cost areas such as youth development activities-academic tutoring, sports, theater, community service, mentoring, etc.-to achieve a more holistic approach. Title XX projects will need to find efficiencies to maintain the level of services in an environment of rising costs. Targets will be set by Spring 2007 after projects report data collected over a six month period between 2006 and 2007.

Is the program purpose clear?

Explanation: The program purpose is to prevent teen pregnancy and sexually transmitted diseases in youth and mitigate negative consequences associated with being a pregnant and parenting teen. The AFL program supports two types of demonstration grants: (1) Prevention grants to develop and test curricula that provide abstinence education designed to encourage adolescents to postpone sexual activity (referred to as Abstinence or Prevention), and (2) Care grants to develop and test interventions with pregnant and parenting teens in an effort to ameliorate the effects of too-early-childbearing for teen parents, their babies, and their families (referred to as Care or Title XX). The AFL program also supports related research.

Evidence: Title XX of the Public Health Service Act (P.L.97-35) (42 U.S.C. 300z) (Title XX) and Section 510(b)(2) of Title V of the Social Security Act (Title V). This demonstration grant program was authorized in 1981 and first implemented in 1982. It was reauthorized in 1985 but has not substantially changed since 1981. The program goals listed in the Office of Adolescent Pregnancy Programs (OAPP) Mission Statement are to: (1) improve behavioral, social and health outcomes among adolescents served in AFL demonstration projects; (2) increase knowledge in the field of abstinence education and service delivery for pregnancy and parenting adolescents; and (3) improve the quality and range of services offered in AFL demonstration projects. These goals are consistent with the legislation and linked to the HHS 5-Year Strategic Plan.

Does the program address a specific and existing problem, interest or need?

Explanation: This program is still relevant to current health risks concerning out-of-wedlock adolescent pregnancy and child bearing, the prevention of adolescent sexual activity and pregnancy, as well as the continued need to promote relevant research and demonstration projects. For example, 34% percent of women become pregnant at least once before they reach the age of 20--about 820,000 per year. Eight out of ten these pregnancies are unintended and 79% are to unmarried teens. Teen mothers are less likely to complete high school (only one-third receive a high school diploma) and only 1.5% have a college degree by age 30. Nearly 80% of teen mothers end up on welfare.

Evidence: Centers for Disease Control (CDC) National Center for Health Statistics June 25, 2003 news release titled "U.S. Birth Rate Reaches Record Low, Births to Teens Continue 12-Year Decline, CDC National Center for Health Statistics May 20, 2004 news release titled "Despite Improvements, Many High School Students Still Engaging in Risky Health Behaviors, and the National Campaign To Prevent Teen Pregnancy (www.teenpregnancy.org) General Facts and Statistics.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The AFL abstinence grant program is similar to two Maternal and Child Health Bureau (MCHB) abstinence programs in purpose (abstinence education as defined under Section 510(b)(2) of Title V of the Social Security Act), targeted beneficiaries (adolescents), and mechanisms (competitive grants). The AFL coordinates with MCHB and the Administration for Children and Families when awarding grant to ensure that the AFL funds will not duplicate other Federal programs at the grantee level. AFL makes inquiries of grantees concerning other sources of State and local funding for abstinence programs to ensure against duplication.

Evidence: MCHB two abstinence programs are Catalog of Federal Domestic Assistance (CFDA) number 93.110, Community-Based Special Projects of Regional and National Significance (SPRANS) (discretionary program competitively awarded) and CFDA 93.235 Abstinence Education (a formulae grant to States which is subgranted to local governments and community based organizations). The FY 2005 President's Budget proposes to move these programs from HRSA to ACF, due to it's expertise in managing programs to assist adolescents.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program design is efficient and effective and there is no evidence that another approach or mechanism would be better for the intended purpose. The flexibility of demonstration grants allows the program to respond to emerging and changing conditions in the field. The program coordinates with HHS partner agencies and others engaged in related efforts to share knowledge and minimize duplication of efforts.

Evidence: Legislation and grant announcements cited in 1.1 above and Office of Population Affairs (OPA) grant processes including application, competitive award process, site visits, and monitoring of participants served show an effective method to efficiently administer these grants. Grants awards are relatively small, averaging less than $270,000 and leverage Federal funding by requiring significant grantee matching and community support. An example of demonstration grant flexibility is grantees finding that the proposed dosage of intervention is not working and grantees working with AFL staff to change the program design. For example, a grantee proposed 10-12 participant sessions; but when the desired changes in knowledge were not occurring, an after school program was added to mentor participants and provide more support to prevent risky behavior.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The award process ensures that grantees understand and agree to the expected program delivery and adolescent target population, and have the capability to meet the program purposes, e.g., serve an area with a high incidence of adolescent pregnancy. In addition, (1) grantee program materials are reviewed by AFL to ensure they are appropriate for the respective target population; (2) training/conference workshops are provided to AFL grantees in how to communicate and effectively reach the target population and involve families; (3) annual reports document the number and demographics of adolescents served; and (4) site visits ensure that the target population as described in the grant agreement are actually receiving the services and there are no unintended beneficiaries.

Evidence: Legislation and grant announcements cited in 1.1 above, AFL grant processes including grant applications and review forms and signed assurances from grantees. End of the year statistics are compiled from grantee reports to track client's served (e.g., in 2003 there were 23,103 Care and 85,363 Prevention clients served). AFL utilizes the Health Resources and Services Administration (HRSA), Shortage Destination Branch information on under-served areas, a list of current grantees, lists of other program office grant locations (MCHB and ACF), and a map to balance the special needs of rural areas and under-served areas in making final funding decisions.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: There are no long-term performance measures that determine overall program outcomes. However, AFL is developing performance baselines, measures, and targets to be used in 2005 grantee reporting. Documentation from the grant application, review, award, and monitoring processes provide a clear and specific description of what is expected from individual grantees such as plan for program design, delivery, goals, expected outcomes, and evaluation.

Evidence: AFL legislation requires each grantee to conduct an independent evaluation that examines the program's effectiveness and progress toward achieving key outcomes with its participants. For example, one Care grantee's goals included reducing by 30% the number of adolescent mothers who have a subsequent unintended pregnancy and this was measured using a pre-, post-test comparison study.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: There are no overall performance measures, targets, or timeframes. However, individual grantees are held accountable. For example, one grantee used pre-test, intermediate post, and six month tests for an intervention and control group to determine whether targets of a 25% increase in awareness in knowledge, awareness, and adaptive behaviors were met in a Prevention grant program.

Evidence: Even though there are no overall baselines or targets, either annual or long term, grant applications/agreements include grantee specific targets and timeframes. AFL uses a combination of factors to set grantee targets: (1) a competitive award process with both internal and external evaluators; (2) staff knowledge and experience about success and failures of prior programs and program designs; (3) specific requirements in program law; and (4) sharing of knowledge within HHS and among grantees through conferences, training sessions, and electronically such as using a ListServ.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: There are no overall annual performance measures, targets, or timeframes.

Evidence:  

Does the program have baselines and ambitious targets for its annual measures?

Explanation: There are no overall annual performance measures, targets, or timeframes.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: There are no overall annual performance measures, targets, or timeframes. However, individual grantees are held accountable for their specific grantee level goals.

Evidence: The program uses the following to communicate and document individual grantee commitments to program goals: (1) grant announcements; (2) grant application package; (3) training sessions for grant application; (4) grant application and assurances; (5) external and internal grant reviewer forms; (6) monitoring by an AFL Project Officer, including annual site visits with a first site visit within 6 months; (7) AFL review and approval of program materials before use; (8) annual grantee conference; (9) regional technical workshops; and (10) end of year progress reports.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent and comprehensive evaluations of the AFL program have not been conducted. AFL legislation requires a limited grantee budget for evaluation (1 to 5%) with the clear stipulation that the evaluating institution must be a local college or university. AFL hires external evaluators to review the evaluations prepared by grantees, and in many cases the evaluation design and implementation has been determined to be inadequate. Conference and technical assistance activities are evaluated by participants and grantee input is implemented into future planning.

Evidence: In June, 2000, AFL's Office of Grants Management was evaluated by an independent consultant. The findings have been utilized in improving program activities and business processes. For example, based on a finding that too many of the grant applications received were of poor quality, the latest Prevention announcement included no-cost technical assistance training to potential grantees at various locations around the country. To improve grant accountability, Grants Management staff now participate in grantee visits.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget resources are not tied to performance goals. Requests are tied to the program's legislative purpose, i.e., Care, Prevention, Research, or combination projects).

Evidence: Budget requests are tied to the program's legislative purpose as illustrated within both the Department of HHS' FY 2005 Office of Management and Budget Justification (OMBJ) and FY 2005 Congressional Justification (CJ).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: A consultant is assisting AFL in developing performance baselines, measures (annual and long-term), and targets to be used in 2005 grantee reporting. A standardized evaluation tools will allow AFL grantees to compare their results with other Care and Prevention programs and national norms and demonstrate progress toward a core set of outcomes. It will allow AFL to report grantee performance using uniform data collection and a common set of indicators. AFL is redeveloping its research and evaluation division and has recently hired a director of evaluations. This will enable a more thorough assessment of AFL program evaluations and permit cross site evaluations. One key activity will be to disseminate AFL evaluative information to the public. AFL is working on a presidential initiative to develop scientifically based standards for an abstinence based curricula. The document is being developed by AFL staff and evaluation consultants with input from national experts including AFL grantees. It will summarize grantee experience and current literature and provide guidance in developing abstinence programs.

Evidence: An interim consultant's report (November 2003) proposed 16 short-term outcomes for Care which provide outcome measures. The consultant is currently working on a similar set for Prevention grants as well as approval under the Paperwork Reduction Act (PRA) for the evaluation instruments. The consultant will also provide assistance obtaining approval under the PRA. Current end of year information collections do no have PRA approval. AFL has recently hired a Director of Evaluations who will be responsible to monitor and work to improve the grantee's evaluation processes. To encourage grantees to enhance their evaluations, the April 2004 grant announcement allows grantees to propose using up to 25% of funding for evaluation-intensive projects.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AFL monitors grantees during the year. Annually AFL collects information on performance as prescribed in the grant agreements. Annual site visits include a written evaluation to the grantee. Grantees are required to respond with a written corrective action plan. AFL staff follows-up and monitors corrective action. For example, a site visit found that a Lifeskills program which requires extensive staff training was being conducted by new staff that were self taught. The grantee was counseled and required to take corrective action. Grantees with financial or other grant management problems are placed on "high risk" status which includes closer monitoring by Grants Management staff including requiring submission of additional documentation prior to release of funds.

Evidence: Grantees provide the following annual information: (1) end of year progress report on program, program evaluation, demographics, and dosage statistics on program activity; (2) expenditure and budget justification; and (3) a continuation application. Project Officers review grantee performance information and write programmatic and evaluative reports which are reviewed by the Director, OAPP. Performance information is used to provide recommendations to grantees, require corrective action and to plan orientation conferences and annual technical assistance workshops.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Grantees are held accountable for cost, schedule, and performance results through monitoring visits, conferences and technical workshops, and annual performance reports. Information obtained is used to determine continued funding, need for additional technical assistance, and any need for performance/delivery changes. Assessment of this information is used for corrective action and continued grant funding is contingent upon demonstrating satisfactory progress. All OAPP employees have performance plans or contracts, assessed by supervisors to evaluate job performance, e.g., ensure that grantee's program elements include specific, measurable, time-framed objectives and are linked to grantee evaluation plans.

Evidence: OPHS has assigned a budget specialist for each grantee to review associated budget requests, reports, and justifications. If a grantee is not able to handle fiscal matters appropriately, they will be placed on 'special status' and will not be able to drawn down funds without prior approval. Grantees are required to file a Financial Status Report, Standard Form 269, within 90 days of budget year end to account for all program expenditures (Federal and non-Federal), program income, and to show that the required matching requirements are met. Contracts are funded under a delivery schedule, whereby payment is not made until the contractor performs. A Project Officer reviews and approves all deliverables on a cost schedule.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program obligates funds in a timely manner and financial statements show minimal unobligated balances. The funds are obligated according to the fiscal year and in accordance with the standards and guidelines of the Department's accounting system and OMB Circulars. Grantees report on planned and actual expenditures. Grantees provide a cash transaction report indicting the drawdown of funds and balances on a quarterly basis. Grantees are required to produce a Financial Status Report (FSR) and reconcile OMB Circular A-133 audits with the FSR. AFL monitors grantee expenditures to ensure compliance with legislation, regulation and policies. Annual site visits are made to each grantee.

Evidence: Regular accounting reports show funds obligated consistent with annual budget and Congressional Justification. Grantees spending over the OMB Circular A-133 threshold are required to have audits. Resolution of A-133 audit findings is coordinated with the Office of Inspector General.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: During the competitive grant award process, AFL reviews closely the relation between grant amount, services to be provided (e.g., number of clients served and level of services to be provided to ensure a cost efficient grant), and timeline to ensure an efficient and cost effective grant. The grantee's end of the year report, application for continued funding, and annual AFL site visit are used to ensure the services are actually being provided.

Evidence: The scope and level of services provided by demonstration projects vary too much to have a standard cost formulae across grantees. However as part of pre-award, the judgment and experience of the staff is used to review cost efficiency. For example, the staff review form specifically addresses reasonableness of estimated cost considering level of service to be provided to clients. Grants management staff reviews both direct and indirect costs.

Does the program collaborate and coordinate effectively with related programs?

Explanation: AFL works with other similar programs to ensure coordination. For example, the Maternal and Child Health Bureau (MCHB) funds SPRANS, a similar competitively awarded abstinence program. Some of the grantees funded under AFL also receive funding from MCHB. The review of grant applications considers other similar funding and AFL works with MCHB to prevent duplication of awards. AFL site visits monitor to ensure no duplicate funding of the same project. The grant award process includes input from the local, State, and Federal levels. For example, grant applications must include letters of community commitment to the project. AFL grants are subject to State coordination under the Intergovernmental Review Requirements. Technical evaluation is coordinated within HHS. Subject matter is coordinated within HHS as well as with grantees and experts in the field, e.g., through conferences and speaking engagements.

Evidence: Under Intergovernmental Review Requirements applicants for the AFL grants must submit copies of applications to their State Governor and to the State Single Point of Contact who may comment to AFL on the application.

Does the program use strong financial management practices?

Explanation: Pre-award considers financial capability of grantees. Site visits include grants management issues. Close monitoring of program by AFL staff minimize the risk of improper payments. OPHS is audited annually as part of the HHS overall annual audit and there are no material weaknesses or other deficiencies reported relative to the AFL program.

Evidence: Staff application review form, site review form, and financial management capability reviews are performed both pre and post award by the OPHS Grants Office. AFL site visits are conducted in conjunction with Grants Management staff.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: An independent evaluation was conducted of the AFL grant process in 2000. Annually the Director of the OAPP reviews the AFL program and workplan and facilitates a one day in-service meeting to review the past year activities and propose improvements for the next year.

Evidence: Examples of recent improvements based on internal and external assessments are: (1) orientation workshops for new applicants, (2) list-serve for grantees to expedite communication and information dissemination between projects; (3) utilizing the simplified noncompeting continuation application process (SNAP) to expedite continuation reviews of 2 - 5th year grantees; (4) allowing grantees to plan and implement their own training workshops to cut costs and improve attendance; and (5) creating a training CD-Rom and Web-Based training for all AFL applicants.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All grants are awarded competitively. The process includes public notice in the Federal Register, technical assistance workshops to help prospective applicants, and an application review process that includes both external and internal reviewers. Grants are for a maximum term not exceeding 5 years (3 years for Research).

Evidence: The awarded process starts with a Request for Applications (RFA) which is published in the Federal Register and clearly states the requirements and review criteria. Through an inter-agency agreement with the Agency for Healthcare Research and Quality (AHRQ), the Health Care Policy and Research Special Emphasis Panel convenes a group of experts in the field to conduct of AFL research applications. Grant applications are reviewed by a pool of external reviewers who are selected based on their expertise in the field and geographic location. During the two day external review meeting, each application is debated, scored by 3 reviewers, and given a recommendation for funding. Next, AFL program staff perform an internal review and make recommendations for funding decisions based upon the external reviewers scores/recommendations. The Deputy Assistant Secretary makes the final decision based on the scores and recommendations of the external reviewers and AFL program staff.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: An AFL Project Officer is assigned to each grantee. The Project Officer monitors the assigned grantees through annual site visits, contacts at regional training meeting and the annual conference, and review of end of the year reports and continuation applications. In addition, each grantee is assigned a Grants Management Specialist who reviews all budgetary activities and participates in selected site visits. Research activities are monitored as part of the annual continuation application process.

Evidence: The oversight is documented in site-visit feed back letter to grantee and reviews of grantee end of the year reports, continuation applications. Grantee reports include programmatic and statistical information which is reviewed by the Project Officer and financial information which is reviewed by the Grants Management staff.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The AFL program does not collect, compile and disseminate grantee performance information or make it available to the public.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: There are no overall long-term performance measures, targets, or timeframes, therefore progress cannot be demonstrated. The AFL programs require that all demonstration grants include independent evaluations. Although these evaluations have provided some evidence of successful programs and interventions at individual grantees, there is no common set of core measures. Core instruments for both prevention and care programs are being developed this year.

Evidence: AFL Request for Applications, applications, continuation applications, program guidance, and conferences provide information on how grantees can achieve performance goals. Annual site visits and frequent correspondence between AFL program staff and grantees provide information on grantees meeting their annual performance measures.

Does the program (including program partners) achieve its annual performance goals?

Explanation: There are no overall annual, performance measures, targets, or timeframes.

Evidence:  

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: AFL does not have efficiency or cost effectiveness measures and targets for the program as a whole. However, AFL's grant award process includes considerations of costs relative to level of service to be provided. Grantees are closely monitored during the grant period. The monitoring includes programmatic and financial issues and a feedback loop to the grantee ensuring that the agreed to service levels and time schedules are met. AFL has used an external consultant and annually uses an internal process to review internal operations.

Evidence: Grantees program design is reviewed by both internal and external evaluators prior to award. Annual site visits, annual reports and the continuation application process is used to monitor grantees. Annually the Director, OAPP reviews the AFL workplan and facilitates a one day in-service meeting to review the past activities and propose improvements for the next year. In June, 2000, AFL's Office of Grants Management was evaluated by an independent consultant. The findings have been utilized in improving program activities and business processes.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: AFL believes their grants are unique (e.g., demonstration in nature which allows flexibility, require an independent evaluation, and include parental and family involvement) and therefore has not compared them to other programs.

Evidence: See Question 1.3 discussion of two similar MCHB programs.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: As stated in question 2.6, independent and comprehensive evaluations of the AFL program or its subparts (care and prevention demonstration projects, or research studies) have not been conducted at the national program level.

Evidence: The AFL program supports the evaluation activities of each funded demonstration project individually but does not provide for overall program evaluations.