Detailed Information on the
Smaller Learning Communities Assessment

Create a mechanism for making program performance data more widely available to the public.

Working with Congress to terminate funding for this duplicative program.

Use program data to establish baselines and long-term annual and targets for performance measures that do not yet have them.

Continue to implement an efficiency measure for the program.

Measure: The percentage of students in high schools receiving Smaller Learning Community grants who meet or exceed the proficient level of performance on State reading assessments.

Explanation:Data are for the 2000 grants cohort, which received 3-year grants.

Measure: The percentage of students in high schools receiving Smaller Learning Community grants who meet or exceed the proficient level of performance on State mathematics assessments.

Explanation:Data are for the 2000 grants cohort, which received 3-year grants.

Measure: The percentage of students in high schools receiving Smaller Learning Community grants who graduate from high school.

Explanation:Data are for the 2000 grants cohort that received grants for a 3-year period.

Measure: The cost per student scoring at or above proficient in reading. (New measure, added 2/07)

Explanation:

Measure: The cost per student scoring at or above proficient in mathematics. (New measure, 2/07)

Explanation:

Measure: The percentage of graduates in schools receiving Smaller Learning Communities grants who enroll in postsecondary education, apprenticeships, or advanced training for the semester following graduation.

Explanation: 

Is the program purpose clear?

Explanation: The purpose of Smaller Learning Communities (SLC) is to award grants to school districts to create and implement smaller, more personalized school learning environments. The statute lists many different activities, besides creating structural changes in high schools, that can be supported with program funds. Recent Department grant competitions have attempted to create more focus for the program by emphasizing activities that have the greatest likelihood of improving student achievement. Also, in 2001 through 2005, appropriations language focused the program more narrowly by directing the Department to make grants only to support the creation of smaller learning communities in schools that include grades 11 and 12 and serve at least 1,000 students in grades 9 and above.

Evidence: Title V, Part D, Subpart 4 of the Elementary and Secondary Education Act of 1965, as amended by Public Law 107-110, the No Child Left Behind Act of 2001. Consolidated Appropriations Act, 2005, Public Law 108-447, Division F, Title III, Vocational and Adult Education account. Notices of application priorities for calendar year 2004 and 2005 grant competitions.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses the problem of student alienation in large schools - from teachers, other students, and learning - that often hampers student achievement. While structural changes and personalization strategies, by themselves, are not likely to improve student academic achievement, they may create valuable opportunities to improve the quality of instruction and curriculum and to provide the individualized attention and academic support that all students need to excel academically.

Evidence: Some empirical studies have found that smaller school size is associated with higher student achievement, particularly among disadvantaged students, but no studies have been conducted using true experimental designs or quasi-experimental designs with rigorously matched comparison groups. Studies that use "smaller learning communities" in large high schools as the unit of analysis are more limited and less rigorous (Bomotti, 2004). Researchers generally assert that the smaller size of a school or a "learning community" with a larger school only promotes improvement in student achievement indirectly by facilitating curricular and instructional reforms, greater student engagement, and more collaboration among teachers. A "small" environment, in and of itself, does not improve student academic outcomes (Lee, Smith, Croninger, 1997; Gladden, 1998).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: SLC is the only Federal program that is exclusively focused on creating smaller learning communities in large high schools. However, in recent years, significant private-sector funding has become available to promote small schools, including a major high school initiative sponsored by the Bill and Melinda Gates Foundation. Although the SLC's funds have supported only the creation of smaller learning communities within large high schools and the Gates Foundation has focused mainly on the creation of new schools, there has been a significant amount of overlap.

Evidence: In 2005, the Gates Foundation reported a total investment of about $1 billion in efforts to improve high schools, including support for the creation of more than 1,000 new small schools in 40 States and the District of Columbia. The Gates initiative has also supported small-school reforms in existing large high schools. /www.GatesFoundation.org

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: While the program narrowly focuses on implementing smaller learning communities, it has design advantages (e.g. flexibility) that have allowed the Department to focus grant competitions to emphasize improved outcomes for students enrolled in programs funded by SLC and also to use program funds to link SLC grants to broader high school reforms through research, technical assistance, and program outreach.

Evidence: Department SLC grant application notices and other materials. Letter dated March 15, 2005, sent to the House Appropriations leadership, responding to the directive, in the fiscal year 2005 House Appropriations Committee report (No. 108-636) requesting an outline of Department plans for the use of FY 2004 national activities for research, technical assistance, and program outreach and networking to promote connections between local grants and broader high school reforms.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program targets funds exclusively to high schools serving at least 1,000 students, which creates a pool of approximately 4,733 eligible high schools. As of August 2005, program grants have been awarded to support the planning and implementation of SLCs in about 25 percent of the eligible high schools. However, many of the eligible schools have not chosen to create smaller learning communities, which may be a sign that the program has already reached a large proportion of the LEAs with eligible high schools that have strong commitment to, and support for, the SLC restructuring strategy. Schools funded through SLC grants are larger than the average large high school (median enrollments of 1,874 students vs. 1,554 in large high schools generally) and have a much higher percentage of minority students (median of 60 percent vs. 22 percent).

Evidence: Applications for FY 2003 funds offer some evidence of decreased demand for program funds (although the situation may change in subsequent competitions). Even with two competitions, the pool of fundable applications was insufficient to absorb the available FY 2003 funds. Rather than fund lower-quality applications, the Department lapsed $26.5 million in unused FY 2003 SLC funds. The Department did not lapse FY 2004 funds; however, interest in the program appeared to be more narrowly concentrated. Approximately one-third of applicants and grantees had received previous implementation grants, and grantees in two States (California and Florida) received about half of the funds available for the competition.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has adopted long-term measures that meaningfully reflect SLC's purpose. These measures include performance on reading and math assessments and high school graduation rates.

Evidence: The measures are included in the Department's Planning and Performance Management Database.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Department has not yet set the required targets and timeframes.

Evidence:  

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The annual performance measures reflect the goals for students participating in smaller learning communities, including students' achievement on State reading and math assessments and graduation rates. After the Department sets long-term targets and timeframes for the annual measures, they will be used to demonstrate progress toward long-term goals.

Evidence: The measures are included in the Department's Planning and Performance Management Database.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Department has baseline data collected in 2001 for the antecedent program and has established ambitious targets through 2006 for the three annual measures (student achievement in reading and math and high school graduation).

Evidence: Department Planning and Performance Management Database.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: In the awards resulting from earlier competitions, grantees primarily focused on implementing structural changes and other strategies to create smaller, more personalized educational environments for students. However, projects funded through the most recent competitions stress additional elements designed to enhance student achievement, and grantees receive technical assistance to implement the new elements. Technical assistance providers and grantees work closely with Department staff on implementation of smaller learning environments designed to help improve student achievement.

Evidence: Before receiving funds, grantees provide data from the three years prior to receiving the grant. Grantees must also identify community stakeholders and demonstrate parent support for program implementation. Grantees report annually on their progress in implementing the activities identified in their applications and achieving the goals and outcomes of the program. Contractors visit each grantee to monitor their progress and to provide technical assistance in resolving impediments to implementation. As part of SLC implementation, grantees and technical assistance providers are focusing on implementing coherent plans to collect and use project data for management and evaluation of program effectiveness. Technical assistance providers play key roles in assessing grantee progress through site visits to address evaluation issues and information needed for each project's annual performance report and to help improve local project evaluations.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Current evaluations include an implementation evaluation, completed in May 2005, that will support program improvements by identifying barriers to implementation of SLCs in high schools, and, with some major limitations, assess effectiveness by examining achievement and other data in schools both before and after schools received SLC funds. The Department used the preliminary results of this evaluation to inform the development and implementation of the FY 2004 SLC grants competition. In addition, the Department's Institute for Education Sciences (IES) is conducting a rigorous study that will, in the future, inform efforts to promote smaller learning communities by examining whether two supplemental reading interventions improve reading proficiency in high schools operating 9th-grade "freshman academies" (a popular approach that schools are implementing to create smaller learning communities). With approximately one-third of all students entering and exiting high school with low-level reading skills, there is an urgent need to identify effective interventions to help young people acquire the reading skills they need to succeed in postsecondary education and the workforce. Evaluation findings will help to guide the structure, content and professional development activities of future smaller learning initiatives by providing evidence-based information on which programs can be built, thereby incurring significant future savings in staff time and costs associated with development and implementation. In addition, each project must also conduct an independent third-party evaluation to provide information that will be useful in gauging the project's progress and identifying areas for improvement.

Evidence: The Department used the preliminary results of the evaluation to improve the FY 2004 grants competition by, for example, extending the grant period from three to five years based, in part, on the evaluation's finding that cohort 1 grantees needed greater time to implement career academies and other SLCs in the upper grades. The final report from the national evaluation of program implementation will be released by the fall of 2005. Reports from the IES rigorous evaluation will be released at a later date. Grantees annual reports on third-party program evaluations are submitted as part of SLC annual performance reports.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Since 2003, the Budget has not requested any funding for this program. Although requests have been tied to accomplishment of a major policy goal (that of eliminating funding for programs that do not reflect an appropriate Federal role or, for other reasons, are not Administration priorities), it has not been tied to accomplishment of the annual or long-term performance goals. However, the Department's budget submissions show the full cost (including S&E) for all programs.

Evidence: Department of Education Budget Justifications, 2002-2006.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has made incremental changes in the program competitions each year since 2001 to link structural changes to instructional reforms and other elements to raise students academic achievement. Also, a decline in fundable applications, which resulted in the program lapsing a significant amount of FY 2003 funds, prompted changes in the program's grant-making and technical assistance strategies.

Evidence: In 2002, the Department began to require recipients of implementation grants to use the funds to implement smaller environments that served all students in the school, rather than create one or two smaller learning communities involving only some students. Since FY 2003, ED has required applicants to implement research-based strategies to address the needs of students who enter ninth grade with reading or math skills that are significantly below grade level. Beginning with 2004, competition priorities focuses on changes that not only address the organizational structure of schools, but also focus on making curriculum more rigorous. Beginning with 2005, the Department instituted changes designed to provide grantees with more time and resources to carry out their plans by, for example, extending the project period from three to five years, increasing the award amounts for individual grants to accommodate additional independent evaluation activities and comprehensive strategies and interventions to assist struggling students, both of which are required activities for SLC grantees.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Department collects and utilizes performance information from a web-based performance reporting system, annual evaluations, and reports from technical assistance contractors. The Department used information from grantees and contractors to structure changes in annual grant competitions to help achieve improved program and grantee performance.

Evidence: During FY 2000 through 2002, the program's performance reporting requirements were driven by the needs of the independent evaluation of the program. The evaluator determined the performance data that would be collected. In subsequent years, the program revised its performance indicators to make them more consistent with the requirements of ESEA Title I, as amended by the No Child Left Behind Act, and the Carl D. Perkins Vocational and Technical Education Act (e.g., measuring placement in employment for those students who do not continue on to postsecondary education), and required grantees to establish specific performance goals for each indicator for each year of the grant period. The program established through rule-making, its authority to take action (including terminating the grant) against grantees that failed to meet their performance goals. (Few other Department discretionary grant programs now agree upon annual performance goals with grantees, and still fewer have established through rule-making their authority to terminate funds if a grantee fails to meet these goals.) During FY 2003 and 2004, the program also implemented a web-based performance reporting system that reduced burden on grantees and made it easier for program staff to use performance data to manage the program and improve performance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: As part of the President's Management Agenda, the Department has implemented an agency-wide system that links employee performance to progress on strategic planning goals. New performance agreements hold managers accountable for meeting established deadlines for awarding grants and for managing their programs through monitoring, technical assistance, and data collection activities that are designed to focus on raising and measuring performance. Also, contractors provide technical assistance to grantees to help improve the quality of project implementation and evaluation, and information submitted as part of annual performance reports.

Evidence: Performance agreements for Federal managers address the five most important performance objectives that are specific to their area of responsibility during each rating cycle, including cost, grant scheduling, and performance. Information obtained from reports, conferences, and technical workshops help the Department set work agendas for technical assistance providers, who are held accountable for work products, especially site visit reports that that detail grantees' project implementation status. Grantees are required to submit annual performance and financial reports and information from independent project evaluations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated within the time frames set out by Department schedules and used for the purposes intended.

Evidence: Department of Education Grant Award Processing System (GAPS) and FY 2006 Budget Justification.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program does not currently have an efficiency measure. However, during the competitive grant award process, staff review closely the relation between grant amount, services to be provided, and timeline to ensure cost effectiveness. Annual grantee's performance reports, independent project evaluations, and site visits help the Department monitor the implementation of cohesive plans to address all elements involved in the implementation of SLC projects.

Evidence: The variety in the scope and level of services provided by the projects makes it difficult to develop a broad cost efficiency measure. Currently, as part of the pre-award negotiation, the program depends on staff judgment and experience to review cost efficiency. For example, staff evaluate the management plan required of each grantee to ensure the adequacy of resources, including the extent to which the budget is adequate and costs are directly related to the objectives and design of the research evaluation and SLC activities.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Several formal collaborative arrangements are designed to help achieve research and networking objectives.

Evidence: Through a collaboration between the Department's Institute of Education Sciences and the Office of Vocational and Adult Education, SLC funds are supporting a 4-year experiemental research study that will evaluate the effectiveness of two promising supplemental reading programs to be implemented within a freshman academy smaller learning community model. As part of this study, the Department is conducting a special competition in calendar year 2005 to select school districts with high schools that will implement these reading programs. Findings from the study will be widely disseminated to inform schools that are interested in using literacy interventions in small learning environments. Also, the Department sponsored regional summits in 2004 that brought together State officials responsible for secondary and postsecondary education, Governors' representatives, and Department staff from several different programs (vocational education, special education, and English language acquisition) to develop action plans for improving the academic achievement of high school students through SLCs and other strategies. In addition, the Department consults periodically with officials from private foundations that are active in supporting high school improvement activities, including the Bill and Melinda Gates Foundation and the Carnegie Corporation.

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of the program. Pre-award consideration assesses financial capability of grantees. Site visits include grants management issues. The Department has a system for identifying excessive "draw downs" of grant funds, and can put individual grantees on probation, which requires ED approval of all grantee drawdowns. Monitoring by Department and program staff minimize the risk of improper payments.

Evidence: No internal control weaknesses have been reported by auditors. In addition, staff monitors grantee obligations and tracks grantee' drawdown of funds through the Department's Grant Administration and Payment System (GAPS). GAPS is used the Department to track the financial activities of a grant from initial obligation of funds by ED, draw down on the funds, and final settlement of grants.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Accountability for program outcomes is a key management objective. Toward that end, beginning in FY 2003, the Department began holding grantees accountable for agreed-upon multi-year performance targets for student academic achievement, high school graduation, and placement in postsecondary education and employment. In prior years, grantees reported only on student outcomes. The Department has also expanded monitoring using contractors to promote early identification and resolution of management and implementation issues that may affect a grantee's performance.

Evidence: Application notices and other materials; grantee files; contractor's work plans; Department of Education Planning and Performance Management Database.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Awards are made on a competitive basis and judged on their relative merits. The process includes public notice in the Federal Register, technical assistance workshops to help prospective applicants, and an application review process that utilizes external reviewers.

Evidence: Applications are selected through open competition. Qualifying applications are peer-reviewed and the highest scoring are selected for awards. The program maintains high standards for fundable applications.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Department oversight of grantee activities is accomplished through the program's monitoring and reporting requirements and through site visits to projects by technical assistance providers.

Evidence: Program staff review the annual performance reports submitted by grantees and maintain telephone contact with project directors throughout the year. Contractors visit each grantee at least once during the grant period to monitor their progress and to provide technical assistance in resolving impediments to implementation. Grantees that are experiencing significant management problems are targeted for additional technical assistance and more intensive on-site consultations with technical assistance contractors.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: While the program collects grantee performance data on an annual basis, not all of the information has been made available to the public. Reporting for the purposes of GPRA is made public, but not information from grantees' annual performance reports.

Evidence: Department of Education Planning and Performance Management Database; FY 2004 Performance and Accountability Report.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Department recently established a long-term performance measure for this program. However, the Department has collected performance data for multiple years for Cohort I grantees, which received 3-year grants beginning in 2000. For that cohort, from 2001 to 2003, data show an increase in the percent of students scoring at proficient or above on State reading assessments and a decrease in the percentage of students scoring at proficient or above on State math assessments. However, data show little change in the mean number of disciplinary actions (suspensions and expulsions). All program data are self-reported by grantees, but not verified by the Department.

Evidence: Department Planning and Performance Management Database.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has not met its annual performance goals under GPRA.

Evidence: Department's Planning and Performance Management Database.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: While the Department has not developed an efficiency measure for the program, it is implementing management efficiencies to improve the grants, including actions to streamline and focus technical assistance by increasing the number of site visits while reducing the length of those visits from approximately 2.5 days to a maximum of 1.5 days per site. Also, an expanded pool of technical assistance providers with a broad range of expertise and capacity will allow grantees to request and receive assistance more specifically designed to their particular needs. The overall focus of technical assistance has also shifted to include strategies to build State capacity that will reinforce and strengthen local program activities. Technical assistance partnerships with organizations such as the Council of Chief State School Officers and the Council of the Great City Schools have recently supported State and school district action planning and policy development institutes for State leadership teams to support and reinforce reforms taking place at the school-based SLC level.

Evidence: SLC National Activity Spending Plans; Technical Assistance Provider Reports

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although similar to the Gates Foundation initiative, the SLC program has put more emphasis on grantee accountability for improved student achievement and on rigorous evaluation of promising instructional approaches to raise student achievement that are implemented in smaller learning environments. Currently, no comparable evaluation or data exist to compare the two programs. In 2005, Gates indicated that its grants at the high-school level would no longer focus on structural change as a first step in fostering high- school reform.

Evidence: SLC emphasizes project accountability for improvement in student achievement, high school graduation, college enrollment, and employment. SLC funds are supporting a rigorous impact evaluation to examine whether two promising supplemental reading interventions improve reading proficiency in high schools operating 9th-grade "freshman academies" (a popular approach that schools are implementing to create smaller learning communities). With approximately one-third of all students entering and exiting high school with low-level reading skills, there is an urgent need to identify effective interventions to help young people acquire the reading skills they need to succeed in postsecondary education and the workforce. Even with differences in emphasis, however, in the past 5 years, the SLC and Gates initiatives have had objectives in common, including, for example, stimulating activity nationwide to create smaller school learning environments; achieving comprehensive (rather than piecemeal) school reform, connecting local initiatives with broader high school reform efforts; and evaluating funded projects.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Past evaluations of career and freshman academies, the most common approaches used by SLCs, have found only modest impacts on student outcomes. An independent evaluation of the SLC program found that a majority of SLC schools successfully implemented structural reforms, but achieved only modest changes in some student outcomes.

Evidence: A random assignment evaluation conducted by MDRC of career academies, a restructuring favored by a majority of grantees, found that these academies increase student earnings after high school, but had no impact on student academic achievement, high school completion, or enrollment in postsecondary education. A less rigorous and still ongoing MDRC study of Talent Development freshman academies found substantial increases in the promotion of first-time, ninth-grade students to the tenth grade and modest improvements in attendance. The independent evaluation of the FY 2001 grantees reported similar findings. The SLC National Implementation Study, completed in 2005, was a comprehensive, descriptive study of the extent to which the first and second cohorts of grantees met the objectives of the authorizing legislation as they implemented SLCs. The study included an interrupted time series analysis of program outcomes, but the design was not sufficiently rigorous to allow conclusions about the program's impact or effectiveness. The study looked at trends in outcomes before and after receipt of Federal funds, but did not use a comparison group or other means of assuring that changes were due to the receipt of SLC funds and not extraneous factors. Abt Associates of Cambridge, Massachusetts, an independent third-party, performed the evaluation. Among other things, the study measured the extent to which schools funded in FY 2001 implemented all of the key features of the SLC program, as established by Congress and ED, by the end of the grant period. Freshman and career academies were defined as either "high implementing," "moderately implementing" or "low implementing" based on whether each had successfully implemented a specific number of defined features, such as common planning time for teachers. Freshman academies created by a majority of schools (46 of 58) were found to be high or moderately implementing. Career academies created by a majority of schools (34 of 44) were found to be high or moderately implementing. Data showed little change in academic and behavioral outcomes, based on annual performance report data, including no significant changes in student achievement, either as measured by scores on statewide assessments or college entrance exams; increases in student extracurricular participation and promotion rates from 9th to 10th grade; and slight decreases in the incidence of school violence, disciplinary action, and alcohol and drug use. The evaluation also notes other trends in outcomes such as increases in the reported percentages of students taking the SAT and intending to continue to postsecondary education.