Integrated Performance Information for Workforce Development A Blueprint for States DRAFT February 2005 Washington State Workforce Training and Education Coordinating Board On behalf of the Integrated Performance Information Project State Teams from Florida, Michigan, Montana, Oregon, Texas, and Washington Principal author: Bryan Wilson, Ph.D. bwilson@wtb.wa.gov This project was funded through Grant # AN 13397-03-60 from the U.S. Department of Labor, Employment and Training Administration, in the amount of one million dollars. The project is funded 100% by this federal grant. This interim report on the project reflects the opinions of the authors and does not represent official opinion or policy of the U.S. Department of Labor. State of Washington Workforce Training and Education Coordinating Board 128 10th Ave., S.W. P.O Box 43105 Olympia, Washington 98504-3105 Telephone: 360-753-5662 •Fax: 360-586-5862 http://www.wtb.wa.gov Email:wtecb@wtb.wa.gov Ellen O'Brien Saunders Executive Director Integrated Performance Information for Workforce Development A Blueprint for States Executive Summary Introduction This is a guide for states interested in creating or further developing integrated performance information for workforce development programs. Integrated performance information reports performance results consistently across programs, across levels (from institutions to local areas to states), or for programs as a system. It responds to the longstanding challenge and frustration caused by multiple, inconsistent performance measures across workforce development programs, a multiplicity that impedes collaboration—in both planning and service delivery—and befuddles policy makers. It also responds to shortcomings in programs' management information systems that cannot follow participants over time or report performance in a consistent manner. Integrated performance information, however, is more than a shared information system and a set of consistent measures. It also requires institutions and practices to support shared accountability for results. This Blueprint discusses each of the steps involved: establishing authority, building a culture of shared accountability and trust, generating capacity, crafting performance measures, setting and using targets, as well as, creating and maintaining a shared information system. Some states are at the initial stage of considering whether they want integrated performance information; others may have been at it for a long time, but are interested in improving their work. In either case, this Blueprint is intended to be of assistance. States may want to consider bits and pieces, or the whole thing, as best suits their needs. The Benefits There are many advantages to states having integrated performance information. They include increased accountability, improved strategic planning, better research, more efficient use of resources, and a sense of shared-responsibility among workforce development programs. These advantages can improve the credibility of workforce programs and, in turn, enhance the support they receive and, ultimately their ability to serve customers. What is meant by workforce development? The phrase, workforce development, encompasses programs that prepare people for employment and career advancement throughout their lives, and includes, but is not limited to: • Secondary Career and Technical Education • Postsecondary Career and Technical Education • The Employment Service, Workforce Investment Act (WIA) Title III • Workforce Investment Act Title I-B • Trade Adjustment Assistance Act • Adult Education and Family Literacy, WIA Title II • Vocational Rehabilitation, WIA Title IV • Temporary Assistance for Needy Families Work Program • Apprenticeship Viewed as a system, it may surprise some to learn that most money for these programs comes from the states. The largest programs, in terms of funding, are the education programs for which states typically supply at least 90 percent of the funds. Given this funding arrangement, it is logical that states exercise leadership in devising integrated performance information across workforce development programs. The United States Office of Management and the Budget (OMB) has taken an important initial step toward integrated performance information by issuing "common measures" for federal workforce development programs. OMB has received the attention of federal agencies, particularly the Department of Labor (DOL), regarding the need for consistent measures across programs. DOL has also taken the step of designing a new reporting system, ETA Management Information and Longitudinal Evaluation System (EMILE), that is to be consistent across most Department programs. This Blueprint builds on these initial steps by recommending performance measures and an information system that would support consistency across state as well as federal workforce programs. Using the Blueprint does not, however, require implementation of a system such as EMILE. The Blueprint was produced through the joint efforts of six states (Florida, Michigan, Montana, Oregon, Texas, and Washington), with the financial support of DOL. Washington State's Workforce Training and Education Coordinating Board (Washington Workforce Board), with the assistance of the National Governors Association's (NGA) Center for Best Practices convened policy and technical teams from each of the six states. Each state team included representation from a cross-section of workforce development programs. This diversity of representation was a necessary ingredient for the success of the project. The team members endeavored to listen closely to the perspectives of each program and to arrive at solutions that were acceptable to all. (Appendix A lists participants.) The state teams met several times during 2004 to share experiences and lessons learned, review technical papers, think through key questions, and arrive at consensus on key aspects of integrated performance information. In addition to the NGA's Center for Best Practices, the states received assistance from the Ray Marshall Center at the University of Texas and The Center for Governmental Studies at Northern Illinois University. The states benefited greatly from the research conducted on behalf of the project as well as from the general expertise and experience of these entities. They and the state teams provided much of the material for the Blueprint and reviewed and commented on drafts. The Blueprint is very much the shared product of the six states and their partners, although the Washington Workforce Board remains ultimately responsible. The Blueprint consists of the following sections: Part I: Challenges and Responses States face serious challenges as they embark down the road of creating integrated performance information. And the challenges don't end with the beginning. Most of the challenges are ongoing and require constant attention. This section of the Blueprint examines some of the major challenges and choices that some states have made to address them. The section discusses: (1) establishing authority for integrated performance information, (2) creating a culture of shared accountability, (3) building capacity-including funding and addressing privacy issues, and (4) reaching consensus on goals and measures. For each challenge, the Blueprint presents examples of actions states have taken, the variety of actions reflecting the institutional structures and political conditions in states. States reading the Blueprint may want to pick and choose from these examples and implement the steps that best fit their situation. Part II: IPI Performance Measures What are the best performance measures for workforce development if the same measures are applied horizontally and vertically within the system, including programs that are funded mostly by the states and programs that are funded mostly by Congress? This was a central question considered by the six states and their partners at NGA's Center for Best Practices and the Ray Marshall Center at the University of Texas. The states began by considering, "What do policy leaders want to know about performance results?" They then suggested the selection criteria for judging measures, and analyzed the advantages and disadvantages of a long list of possible measures. In the end, the states agreed on a relatively short list of measures that best respond to the performance questions commonly posed by policy leaders. This section of the Blueprint summarizes the discussion and recommendations of the six states' teams. Other states may wish to follow suit, either by following this type of process within their state, or by adopting some or all of the recommended measures. The following table summarizes the performance measures recommended by the teams from the six states. The measures are separated into those measures that are useful as accountability measures, for which there could be targets and consequences, and those measures that, while indicators of how well the workforce development system is doing, do not sufficiently satisfy the criteria for good performance measures in order to be used for targets and consequences. TABLE: IPI Performance Measures Accountability Measures Category Measure Labor Market Results for Program Participants • Do people get jobs? • What are they paid? 1. Short-term Employment Rate: The percentage of participants who are employed during the second quarter after exit. (For youth, enrollment in education counts as well as employment.) 2. Long-term Employment Rate: The percentage of participants who are employed during the fourth quarter after exit. (For youth, enrollment in education counts as well as employment.) 3. Earnings Level: Median earnings during the 2nd quarter after exit among all exiters with earnings. Skill Gains • To what extent do education levels increase? 4. Credential Completion Rate: The percentage of exiters who have completed a certificate, degree, diploma, licensure, or industry-recognized credential during participation or within one year of exit. Results for Employers and the Economy • Are we meeting the needs of employers? 5. Repeat Employer Customers: The percentage of employers who are served who return to the same program for service within one year. Performance Indicators Category Measure Results for Employers and the Economy • Are we meeting the needs of employers? 6. Employer Market Penetration: The percentage of all employers who are served during one year. Return on Investment • What is the return on the investment? 7. Taxpayer Return on Investment: The net impact on tax revenue and social welfare payments compared to the cost of the services. 8. Participant Return on Investment: The net impact on participant earnings and employer-provided benefits compared to the cost of the services. END TABLE The measures do not attempt to measure everything that is important to each program. These measures concentrate on outcomes important across workforce development programs. An individual program may have other goals and measures related to its particular mission. For example, adult education (WIA Title II) has a goal of improving literacy skill levels and measures linked to that outcome.1 Each program may want to have an additional measure or measures related to its unique mission. The state teams considered many other measures besides these eight. This section discusses some of these measures and why they did not rise to the top of the list. This section also discusses data sources, setting and using performance targets, adjusting targets or results for economic conditions and participant characteristics, and how national research complements state performance measurement. 1 Participants in Adult Education and Family Literacy who are in the program for a reason that is not work-related, may be excluded from the performance measures presented here. Part III: Shared Information Systems A major barrier to integrated performance information is the absence of a shared or integrated information system for workforce development in most states. Participant information remains scattered about in various program level Management Information Systems (MISs), and there is usually no system for integrating the information from the multiple MISs to support cross-program performance measurement and reporting. This section of the Blueprint discusses the steps and decisions involved in establishing a "data warehouse" that links administrative records from multiple programs with other data sets containing outcome information, such as unemployment insurance wage records. A data warehouse is built on top of existing MISs and does not replace them. The MISs are still necessary for program management. The warehouse is a longer-term repository where data are cleaned and matched in order to analyze and report performance outcomes and to conduct research. Creating a data warehouse requires states to make decisions regarding: authorization; leadership; funding; scope; data ownership, confidentiality, and access; information flows; reporting; and quality assurance, among other issues. This section of the Blueprint walks through each of these issues and the major options. Conclusion States will want to consider how to use this Blueprint given federal initiatives in this area, including the OMB's common measures, the DOL's EMILE System, and the pending reauthorization of the Workforce Investment Act (WIA), the Carl D. Perkins Vocational and Technical Education Act, and other federal acts related to workforce development. States may choose to proceed in a number of different ways. The development of a shared information system is something that should be useful regardless of the outcomes of federal initiatives. Electronically linking records from multiple programs with files containing outcome data will facilitate the implementation of common measures and reporting the performance information likely to be required by the reauthorized acts, as well as meeting state-identified needs. If states find some or all of the performance measures recommended here to be useful, they can implement them as additional measures to those necessary to satisfy federal or other state requirements. Experience has shown, moreover, that federal performance measurement requirements evolve over time. If states find the IPI measures to be useful in responding to policy makers' needs, the measures may be reflected in future generations of federal acts and guidelines. Federal performance requirements have a powerful effect on program implementation and results; if states find the IPI measures to be useful, it would be very helpful if future federal requirements were aligned with them. Finally, whatever the specific course of events in Congress or the federal agencies may be, the basic issues of building the capacity for and a culture of shared accountability are likely to be challenges that remain with states. The state teams believe the experiences and lessons shared here will help workforce development leaders as they continue to face these challenges.