HomeAboutCatalogCAPSJoin CAPSLinksNewsContact Us

Letter to the Department of Education on School-to-Work Progress Measures

(Our federal advocacy remains an integral, unique and important part of our work. Often it serves as the entry point for helping to change practice where it counts, at the classroom level. In Spring of 1997, we notified the National School-to-Work Office (along with the Departments of Education and Labor) to let them know that we had concerns about the effectiveness and clarity of that office's national progress measures, required by Section 402 of the School to Work Opportunities Act. The letter below led to a series of meetings with the School-to-Work Office and their representatives. CLE is now playing a direct role in the School-to-Work Office's regional trainings to strengthen its evaluation of programs to help schools deliver higher standards and wider opportunities for more students.)

 

21 April 1997

Mr. Patrick J. Sherrill
Department of Education
600 Independence Avenue SW
Room 5624 Regional Office Building 3
Washington, DC 20202-4651

 

Deliver by regular mail, by fax to (202) 708 8196, and by e mail to pat_sherrill@ed.gov

 

Re: Proposed Information Collection on School-to-Work Progress Measures

 

Dear Mr. Sherrill:

Thank you for the opportunity to comment on this matter. In its almost thirty years of existence the Center for Law and Education has sought to improve educational outcomes for children often left behind by the educational system. Our Vocational Education Project works with schools to bring about this end. We are approved by the Department to provide technical assistance to School-to-Work (STW) grantees. We have been actively involved in drafting vocational education and school-to-work legislation. Our comments are informed by significant experience and expertise.

In preparing our comments we have reviewed the Justification packet supporting this data collection and supporting papers cited in it. We also have reviewed the surveys to be used to perform the national evaluation envisioned by the STW Act and discussed in the Justification. The documents we reviewed, taken separately or together, do not comprise a system of data collection and assessment which complies with Section 402 of the Act.

A. Section 402 of the STW Act requires each grantee to provide information from which outcomes can be discerned and quality and equity can be judged

Section 402 requires the Secretaries to establish a "system of performance measures for assessing State and local programs." The performance measures must assess:

  • progress in development and implementation of state plans, and in particular a focus on meeting the requirements of Title I concerning the program components [402(a)(1)]; demographic characteristics of participants [402(a)(2)]; progress in meeting the needs of students and dropouts [402(a)(3)]; progress in meeting state goals to ensure opportunities for women, including participation in nontraditional occupations [402(a)(4)]; outcomes, disaggregated by demographic characteristics, in: academic learning gains, staying in school and attaining :
    • a diploma,
    • a skill certificate,and
    • a postsecondary degree,
  • placement and retention in further education or training, job placement, retention and earnings, attainment of strong experience in and understanding of all aspects of the industry the students are preparing to enter [402(a)(5)] and; the extent to which the program meets the needs of employers, 402(a)(6).

Congress envisioned a system in which each grantee would report information allowing judgments about performance to be made. The things Congress required to be assessed demonstrate its clear concern for quality and equity, judged both by implementation of key parts of the Act and by outcomes.

 

B. Key Concerns

Representative Goodling, Chair of the House Education and the Workforce Committee, notes that STW money is not free; the quid pro quo is accountability. A management axiom says that what does not get measured does not get done. Implication and experience lead us to conclude that, under the proposed system, the job of educating STW students will not get done. We cannot countenance this outcome.

1. The proposed progress measures, whose "strategy, focus, and nature" were defined by the states (Justification, page 6), address precious few of the criteria of Section 402. They reflect issues that states and localities "indicated that they wish to monitor." The progress measures purport to address only three of fourteen enumerated subparts of Section 402 (a): participation of students and employers, (a)(2); high school graduation, (a)(5)(B)(I); and placement (but not retention) in post-secondary education (a)(5)(E).

Completely left out are all the other critical measures above that Congress required to be a part of the system of performance measures. For example, and particularly damaging to effective implementation, improvement, and oversight of school-to-work programs, are the complete omission of measures concerning:

a. Attainment of strong experience in and understanding of all aspects of the industry the students are preparing to enter [(a)(5)(C)].

We are especially concerned with the failure to include this required measure. This key concept, which is pervasive throughout the Act, allows students to gain the broad knowledge and skills necessary to fulfill the purposes of the Act. The government has invested literally billions of dollars in this concept. Since 1990 every grantee under the Perkins Act has been required to teach this concept and to assess the effectiveness of the teaching.

We would consider failure to measure include outcome measures on this (as well as progress implementation measures under (a)(1) above), if not remedied, to be a most serious violation of the Act. If, on the other hand, the National Office wants to remedy this omission, we would be most happy to work with you immediately to develop good measures on an expedited basis, since this is an area where we have considerable implementation expertise.

b. Whether all students are achieving quality outcomes in the key outcome areas identified by Section 402 [a)(5)(A)-(E)].

No effort is made here to include in the system of performance measures any of the required outcomes for participating students and dropouts, broken out for the population groups Congress required. Most of these outcomes -- including academic learning gains -- are not addressed in any fashion. In the one of the two exceptions, immediate postsecondary transition, there is no provision for reporting by the categories of students required by the Act. In the other
exception, high school graduation, disaggregation is requested, but respondents are free simply to say they do not have the data, despite a clear requirement to include it in their reports. (See also the point below.)

c. Development and implementation of state plans that meet the various key program component requirements in Title I, such as (i) a program of study meeting the same academic content standards the State has established for all students and sufficient to prepare students for postsecondary education; (ii) instruction in all aspects of the industry in both school-based and work-based components; or (iii) regularly scheduled evaluations involving ongoing consultation and problem solving with students and dropouts [(a)(1)].

Assessing the extent and quality of program implementation of key components is critical for the school-to-work programs in every state -- both to stay on track and focus implementation, improvement, and oversight before the full impact of the programs will show up in outcome data and to help determine the programmatic changes that are needed to respond to weaknesses revealed in the outcome data.

2. Even the few required areas included at all in the proposed progress measures are not addressed adequately.

a. This is often because respondents are free to decline to provide data that is clearly required, including:

  • demographic data on participating students;
  • demographic data on participation in career exploration and in school-based learning and work-based learning;
  • demographic data on school based learning participation;
  • high school graduation rates;
  • transition to postsecondary education; covers only immediate transition, and at that the data are requested but not required.

As with every element in Section 402(a), the system must provide for assessing each of these elements, and, under Section 402(c)(1), each State is required to provide the information on each element. Saying "we don't know" or "we don't collect it" is clearly contrary to the Act and frustrates its most basic goals.

Examples of other inadequacies are:

b. Overstating opportunities for quality internships by allowing multiple counting of serial internship positions, regardless of duration.

Under the example given in the proposed measures, a student who does nine months of internship (a school year) with one employer could be counted as a single internship in one program, but could be counted as a "series" of nine internships in another.

c. Asking for data in a way that is likely to produce highly unreliable, inconsistent results.

For example, in asking for participation rates in various school-based learning activities, the components are described in ways that would permit a vocational class to be included if there is a single use of addition or subtraction--treated no differently from a class jointly designed and taught by vocational and mathematics teachers to meet National Council of Teachers of Mathematics standards.

d. Asking for only a portion of what is required -- for example asking for immediate post-secondary placement, but not retention.

3. These problems are not remedied by considering items in the national survey.

First, many of the key elements are not included in the national survey either. For example, student outcome measures of attainment of strong experience in and understanding of all aspects of the industry, or of academic learning gains, are not included.

Second, where data is sought, it is often inadequate -- for example, dealing with dropouts by a yes/no question about whether they are served.

Third, questions in some key areas are asked, whether adequate or not in terms of content, only on instruments that are limited to partnerships in a minority of the States. This is completely contrary to the Act's requirement that every state submit reports on each of the elements in Section 402.

This relates to a larger reason why reference to the national survey for the missing data is not adequate. As the proposed collection notes, the national evaluation is designed to serve a very different purpose than the progress measures -- in providing national evaluation data for Congress and public about the overall implementation and success of the program. It cannot fulfill the equally critical purposes of providing partnerships, states, and local communities with the information they need to monitor and improve the implementation of their particular programs.

We support the desire to avoid duplication of data collection efforts. One rather obvious way to avoid this would be to design the system of performance measures, required to assess all programs under Section 402(a) and then, in designing a national evaluation, decide which data from the performance measures will be useful and what additional studies, if any, are needed. This would be consistent with the terms of Sections 402(a), (b), and (c). There may be other approaches, but the bottom line is that there must be a system of performance measures for assessing all programs in all of the required areas, in ways that can be used by the States and programs for overseeing and improving implementation. Instead, we are confronted with a proposal that has gaping holes in almost every major area that is of key concern for implementation of the Act.

 

C. Knowledge of all aspects of an industry is a critical measure of effectiveness

One outcome required by Section 402(a)(5)(C) is attainment of strong experience in and understanding of all aspects of the industry the students are preparing to enter. At page 2 of its Justification the Department cites two papers, dated January and December of 1995, dealing with designing and implementing STW progress measures. The January paper says that assessment of education in all aspects of the industry is a particularly difficult task because few states had developed appropriate assessment instruments (page 7) and that doing so presented "significant challenges" (page 8). We do not dispute the fact few states have developed appropriate measures. However, this is anything but a reason not to develop them as part of the performance measures here. Again, (as in other areas) we would be happy to assist on an expedited basis.

First, as noted above, this is not optional but clearly required by Section 402, and we would view failure to include it in the final document to be a serious violation.

Moreover, the fact that it has not been developed by states should be viewed against the backdrop of the Perkins Act (as well as the STW Act), which has required, since 1990, that (a) States assess all their programs in terms of their capacity to provide strong understanding and experience in all aspects of the industry and then design the state plan to address the needs found in the assessment, and (b) every local Perkins recipient conduct an annual evaluation of the extent to which the program is providing its students with strong understanding of and experience in all aspects of the industry. Apparently, billions of dollars of federal funds have been spent without complying with these conditions. That situation should be remedied now, not compounded by further neglect.

This is particularly true in light of the importance of this element. We work with schools that are educating students in "all aspects of the industry." The idea is proving a powerful catalyst for change.

At the Oakland Health and Bioscience Academy students learn not just technical skills needed to fill many professions in the health sciences, but also how health care is managed and financed. A student might work not only with a nurse in a neonatal intensive care unit, but also with the planner who projects needs, the bookkeeper who processes insurance requests, and the manager who works on financing the purchase of new equipment. The student leaves school with an understanding of the many opportunities in the industry and an understanding of skills which are necessary in that industry but also are central to others.

At Madison Park High School in Boston, what was a legal secretary program is now a program about the law office. Instead of learning narrowly about how to type legal documents, students learn about all aspects of the legal system. They observe at the courthouse. They ride with police. They shadow lawyers and probation officers as they go about their jobs. Resultantly, students have a better understanding of the legal system. They develop broad skills which are transferable across different jobs. The academic content of their vocational courses is more challenging. Teachers have higher expectations of them and they have higher expectations of themselves.

Measuring whether students attain a rich understanding of all aspects of the industry they are planning to enter is an irreplaceable way to discern whether a STW program fulfills the Congressional purposes of making STW systems part of comprehensive school reform and of preparing students for high-wage, high-skill careers of for further education. See Section 3(a)(1). The measurement must be made.

 

D. Congress directed that STW programs measure educational outcomes for all students

Congress expressed its purpose that "all students" be offered high quality school to work programs, Section 3(a)(1)(C), and it emphasized that it meant "all students" to include all students:

"The term all students' means both male and female students from a broad range of backgrounds and circumstances, including disadvantaged students, students with diverse racial, ethnic, or cultural backgrounds, American Indians, Alaska Natives, Native Hawaiians, students with disabilities, students with limited-English proficiency, migrant children, school dropouts, and academically talented students." Section 4(2). Cf. Section 3 (a) (7), (11), (13)."

In the Committee Report accompanying the STW Act the House Committee on Education and Labor attached special importance to minority youth participating in STW programs "in order to close the gap in educational and employment outcomes for these communities." House Report 103-345, p. 31.

The Senate Committee unambiguously asserted similar concern about young women entering non-traditional occupations:

"Ensuring that all young women have the same opportunities, encouragement, and options as young men is an integral component of this Act. The Departments should maintain a statistical breakdown of girls trained in nontraditional occupations. The breakdown should include the occupation and wage-at-placement figures. In addition, the Departments should assess whether proactive measures have been taken by School-to-Work Opportunities programs to recruit, train, place, and retain young women in nontraditional occupational skill areas." Senate Report 103-179, p. 20 (emphasis supplied).

The Senate was perhaps even more emphatic about students with disabilities:

"[U]nder this legislation, students with a full range of disabilities must be an integral part of all aspects of the School-to-Work systems, including career exploration and counseling, planned programs of study and job training that lead to the award of a skill certificate, and data collection and analysis regarding the post-program outcomes of all students.

"The Committee intends that the exclusion of students with disabilities from any aspect of State or local school-to-work systems is unacceptable...

Furthermore, all students, including students with disabilities, must be part of the system of performance measures, and the national evaluation, and data from students with disabilities must be included in any performance outcome evaluation system and reports." Senate Report 103-179, pp. 24-25 (emphasis supplied).

Congress made these concerns concrete in Section 402 of the Act when it required assessment of progress in serving these students. Congress mandated assessment of outcomes, disaggregated by population, including:

  • academic learning gains;
  • high school and postsecondary diplomas received;
  • skill certificates received;
  • placement and earnings after leaving school; and, very significantly,
  • attainment of strong experience in and understanding of all aspects of the industry the students are preparing to enter. 402(a)(5).

As discussed above, the progress measures proposed by the Department do not require reporting of this information.

In addition, as noted above, the proposed survey forms, at Questions VI and VII, seek demographic information about students participating in school and work based learning. In familiar fashion the forms present a grid to be completed, yielding disaggregated data on defined populations (except dropouts and migrants, categories which Section 4(2) of the Act requires). In unfamiliar fashion, however, the forms indicate that the information need not be provided. If it is not available the respondent simply may check a box so indicating and move on. This is inconsistent with the command of Congress.

The national survey would collect only very limited disaggregated data regarding graduation rates and certifications. There is no system for assessing performance.

 

E. The Department's projections of costs are overstated

The Department justifies non-compliance with Congressional direction by citing the "prohibitive costs associated with asking states to develop new data collections to support school to work progress measures exclusively." (emphasis added). The Department cites Oregon's request for $500,000 to develop a new management information system (MIS) for school to work as an example of the cost to the states of collecting the information required by Section 402, concluding that meeting all state MIS needs "would require funds far exceeding the resource capacity of the National School-to-Work office."

A discussion with the author of the Oregon proposal makes clear that $500,000 represents the amount of additional money needed by that state not only to manage STW information but to create a comprehensive MIS for all purposes. She explained to us that in response to a request from the Department for information about costs of data management, she canvassed her state, asking schools to tell her how much money they needed to purchase hardware, software, phone lines, and training for staff to create a MIS capable of handling request for information required by Section 402. She added the responsive figures to arrive at the $500,000 total.

She noted that a comprehensive MIS can be used for purposes beyond managing information. For example, matching students to job placements is difficult. Schools in Oregon which do have computers and Internet access allow students to search work-based learning opportunities on the Internet. Matches of students and job placements are conveniently loaded to a state database.

She also noted the general desirability of a comprehensive MIS. Tracking individual students is becoming necessary for many purposes aside from STW. Such tracking requires assigning unique identifying numbers and entering them into a manipulable, statewide database. She estimated that approximately ten states, including Florida, Michigan, and West Virginia now have this capacity and that several others, including Oregon, are moving towards it.

She was candid about the condition of local data collection, noting that Oregon has a share of people "whose data system is in a shoe box on a secretary's desk." Even so, these people have more trouble with aggregation and transmission of data than they do with collection.

In the modern world, the shoe boxes will need to disappear. The requirements of Section 402 are just one more thing that will lead to their demise. Indulging their continued existence will do nothing to improve the quality of education. Interestingly, despite the existence of shoe boxes, she plans to keep disaggregated data on special populations involved in STW activities.
She believes Section 402 requires it.

 

F. Progress must be measured or it will not occur

We end with our original admonition: progress in serving all students and the outcomes all students achieve must be measured. The law requires it. So does a fundamental allegiance to the principles of quality and equity.

We thank you for this opportunity to comment. We would be happy to supplement our remarks and we hope to discuss these matters with you further.

Very truly yours,

John Vail, Attorney at Law
Education Advocate

Paul Weckstein, Attorney at Law
Co-Director

 

cc: J.D.Hoy, Director, National STW Office
Kristin Conklin, STW Office
Patricia W. McNeil, Assistant Secretary for Vocational and Adult Education

HomeAboutCatalogCAPSJoin CAPSLinksNewsContact Us