NAEP and NCLB Testing: Confirming State Test Results

The National Assessment of Educational Progress (NAEP), also called "The Nation's Report Card," is the only national and continuing assessment of what America's students know and can do in various academic subjects. NAEP administers reading and mathematics tests every two years, and tests other subjects in alternate years. This widely respected testing program is used by national and state policymakers to evaluate the condition of education in the United States.

Under the federal "No Child Left Behind" law (NCLB), NAEP has a new role in efforts to improve student achievement. While state participation in NAEP testing previously had been voluntary, NCLB now requires all states to participate annually in NAEP 4th and 8th grade reading and math tests. However, the NAEP scores are not included in measuring whether schools make NCLB's "adequate yearly progress"  -- the annual improvement required in the percent of students who score at the proficient level on state tests. No consequences to states or schools occur based on NAEP scores

The law thus views NAEP as an independent measure of a state's success in meeting NCLB's goals. Because the NCLB requirement for "adequate yearly progress" allows each state to set its own standards and decide which tests to use, and because state tests and standards vary widely, NAEP will provide a national benchmark so the public can see how students in their state do on state tests compared to the NAEP. If there are wide differences (as there are in most states), then NCLB envisions this creating a dialogue within the state about why such differences exist.

Key Questions about NAEP and ESEA

Given the importance that will be placed upon NAEP test results under NCLB, here are some questions to consider:

• Has your state participated in either the long-term trend NAEP or the state NAEP? Contact your state department of education or visit its Web site to find out. 
• How has a sample of your state's students performed on previous NAEP test administrations in reading and mathematics? This information can be obtained in the reports from NAEP's Web site or from your state department of education. How does NAEP performance compare to your own state's tests?
• If your state does not perform as well on NAEP as it does on its own tests, obtain the NAEP frameworks for reading and mathematics from the National Assessment Governing Board (NAGB) Web site and review the skills and knowledge contained in your standards and tests in comparison to the NAEP frameworks.
• Do the NAEP content frameworks contain significantly different skills in reading and mathematics from your state's existing standards and curriculum? Are modifications necessary to your state content standards?

Because of its new role in NCLB, NAEP will likely begin to influence standards, curriculum and instruction in every state. This raises several policy questions:

• What is or what should be the overlap of content and knowledge between what's on the NAEP test and what's on the individual state tests and content standards? Currently the overlap varies among states. Thus, the NAEP and a particular state test may not be measuring the exact same skills. NAEP scores in a particular state may be higher or lower than the state test depending on how much the tests vary.
• Performance standards vary among the state tests as well in comparison with NAEP. NAEP results are reported by three achievement levels -- basic, proficient, and advanced. These achievement levels may or may not correspond to those used in a state assessment program. NAEP achievement levels or performance standards are widely acknowledged as high standards.
• NAEP is not a "high-stakes" test for students because there are no direct consequences associated with their performance. NAEP leadership has long been concerned about students' motivation to do their best on NAEP tests, particularly at the 12th grade level.
• NAEP is administered in the spring, while some states do their assessments in the fall. That means that NAEP results will reflect almost a year of student growth when compared to those states' assessments.

The Basics on 'The Nation's Report Card'

NAEP is a congressionally mandated project of the National Center for Education Statistics (NCES). NAEP is governed by an independent board called the National Assessment Governing Board (NAGB). This independent board administers the program through test contractors who develop and carry out the assessments.

Since 1969 NAEP has been regularly conducting assessments of samples of the nation's students attending public and private schools at the elementary, junior high and high school levels. NAEP assesses reading and mathematics every two years in grades 4 and 8. In the intervening years NAEP assesses additional subject areas such as science, writing, foreign languages, history, geography and the arts. Participation in these subject assessments remains voluntary under "No Child Left Behind."

Each state selects a representative sample of students to participate. An intact classroom of students is the sampling unit. Your state department of education staff along with a national cadre of testing staff distribute and administer the test. NAEP tests in reading and mathematics are approximately 30-60 questions long. Items on the test are usually spiraled; that is, no one student takes the complete test or even the same set of items that another student may take. Statewide, sufficient numbers of students take all forms of the test so that state profiles can be established. Sampling and test development techniques make individual results as well as individual district and school results impossible. Current law also forbids NAEP from producing individual student results.

Two NAEP programs exist: the long-term trend NAEP and the state NAEP. The long-term trend NAEP assesses the same content established in 1969 in order to provide a consistent picture of student achievement over the years. Long-term trend NAEP is administered periodically to a national random sample of students. State NAEP is administered on a state-sampling basis and assesses updated content to reflect current curriculum priorities.

Testing Plus: Real Accountability with Real Results

As we enter into a national debate on school improvement and greater public school accountability with a heavy emphasis on testing, educators are concerned that a solitary focus on testing ignores important opportunities to help all students achieve at high levels. Over-reliance on testing could have the unintended consequence of hurting more than helping.

NEA proposes Testing Plus -- well-crafted accountability measures that gauge and promote real achievement. The issue is not -- to test or not to test. Students are subject to an array of standardized and teacher developed tests each year. The key question is, "How do we help students achieve, rather than hurting them?" Instead of just applying more tests, NEA calls for smarter testing that also provides students and schools the tools they need to succeed. Such a program includes:

More thorough measures. Schools are complex organizations. It takes improvement in many areas to make an effective school. Multiple indicators (dropout rates, absenteeism, number of students taking advanced placement courses, parental involvement, etc.) would help guide progress toward becoming an excellent school. 

Improved tests and assessments. While the state of the art in testing has progressed tremendously, they remain less than perfect measures of student or school progress. A new generation of instructionally supportive tests is needed. These tests need to be developed in cooperation with teachers and based upon the essential standards such that test results become more useful to the teacher in assisting each student achieve those standards. Classroom assessment practices including the use of portfolios, projects, and performance assessments should also be enhanced through professional development and included in the comprehensive state assessment system.

Comprehensive reporting to parents. Currently, school report cards and pupil report cards are the primary methods of communicating with parents. We call for the expansion of school report data to include (and use) information on the multiple indicators of success and not rely solely on standardized test scores. This could be supplemented by regular reports based upon teacher observation, judgment, and student work samples.

Alternatives to a test as the sole means of accountability. Examples of other accountability systems include school accreditation, visiting teams, and displays of student work to the public. States, school districts, and schools should use multiple indicators of school effectiveness to make improvement decisions at each level. Test scores give little data to improve school operation. 

Targeted professional development. In order to succeed in a standards-based program, teachers need continued professional development targeted toward specific skills and knowledge. 

Focused investments. Teachers report inadequate availability of instructional materials that are aligned to the standards. Focused investments on materials, professional development, and supplemental programs of instruction are necessary to assist teachers and students. 

Testing Checklist: Real Accountability -- What Does It Take?

Testing Plus
Real accountability incorporates the tenets of Testing Plus defined above. Testing Plus provides a broader picture of student progress by using multiple indicators of learning. Most important, Testing Plus provides students, parents, and teachers with the support and resources required to meet higher standards. A school system that is not accountable for providing continuous, high quality resources for school communities has no business holding students and teachers accountable for performance on tests.

Protection Against High-stakes Decisions Based on a Single Test

A nationwide survey conducted by Education Week shows that 11 states identify low-performing schools solely on the basis of test scores. Decisions that affect individual students' life chances or educational opportunities must not be made on the basis of test scores alone. As New York Times columnist Richard Rothstein notes—a baseball player's batting average is never computed based on one game. Accuracy requires that students have multiple opportunities to pass any test when the test results are used to make high-stakes decisions, such as promotion to the next grade or graduation from high school. More importantly, when there is valid evidence that a test score may not accurately reflect a student's true proficiency, predetermined alternatives to demonstrate ability to meet standards should be available to students. Absent such protections, school districts have suffered high dropout rates and degradation in the quality of curriculum. "We could realize significant progress in public education if proponents of standards-based reform joined hands with critics of high-stakes testing and effectively outlawed the use of high-stakes tests as sole indicators of student success," says Panasonic Foundation executive Scott Thompson in a Phi Delta Kappan article.

Adequate Resources and Opportunity To Learn

It's important that the testing cart not be placed before the curriculum and opportunity-to-learn horse. When content standards and tests are introduced as a reform to improve current practice, teachers must have ample opportunity to access professional development and appropriate resources, before schools, teachers, or students are sanctioned for failing to meet the new standards. According to a National Assessment of Title I study, only 40 percent of the schools identified as needing improvement last school year reported receiving additional teacher professional development or other help. Extra learning opportunities and remediation programs for students are imperatives to achieving true accountability.

Clarity in Passing Scores and Achievement Levels

Because the law permits each state to create its own accountability system and its own definition of progress, huge differences in the numbers and percentages of schools identified as low-performing exist across states. For example, Texas identifies only 1 percent of its Title I schools as "in need of improvement," while Michigan identified 76 percent. Worse, many schools are unaware that they have been identified as low-performing. Only four in 10 principals of schools identified as needing improvement reported their status as such. 

The purpose and meaning of passing scores or achievement levels must be clearly stated and understood. Terms like "passing" and "proficient" must be clear and defined according to specific goals. Setting scoring levels such as "minimum competency," "grade level achievement," and "world-class" should be based on educational principles. The consequences of failing to meet levels should be clear.

Coordinated System of Tests and Assessments

Tests that are valid for one use may well not be valid for another purpose. Each separate use of a high-stakes test for individual achievement, school evaluation, curriculum improvement or any other purpose must be evaluated in order to determine the strengths and limitations of the testing program and the test itself. In addition, before simply adding another required test for all students in all schools, a systematic inventory of current testing programs and tests should be conducted. The information gathered would enable the development of a coherent coordinated system that will routinely let teachers, students and parents know how students are doing academically and provide teachers, school administrators and elected decision-makers with accurate data upon which to base policy.

Full Disclosure of Negative Consequences

Before imposing new required testing, policymakers should be aware of the likely unintended negative side effects of any given testing program. Test developers and users have a responsibility to explain the possible harmful effects in all cases where solid scientific evidence exists that a given type of program may produce undesirable results, such as higher dropout rates. It is essential to assure ongoing evaluation of both intended and unintended consequences. Fairness suggests that the governmental body that mandates the test should also provide resources to help all kids meet high standards.

Alignment of Tests to Standards

Tests must be better aligned to essential standards in order to produce valid and reliable results. In its Quality Counts 2001 study, Education Week reports that an analysis by Achieve, a nonprofit group based in Cambridge, Mass., shows that the state standards and tests are not closely enough aligned. Current state tests "tend to measure some standards but not others and to emphasize the less demanding knowledge and skills in state standards."

Recognition of Differences and Disabilities

In the interest of assessment accuracy, testing programs must take into account student differences. For students who are learning English, a test written in English becomes, to one degree or another, a test of language proficiency. The degree of English language proficiency must be considered in deciding to administer the test. In testing students, the effects of their differences must be appropriately weighed in drawing conclusions from the test results.

Explicit Rules for Determining Students To Be Tested

There must be clear policies identifying which students are to be tested and under what circumstances students may be exempted. Without such policies, there cannot be any meaningful comparison of schools, districts or states when changes are tracked over time. The American Education Research Association states, "Such policies must be uniformly enforced to assure the validity of score comparisons. In addition, reporting of test score results should accurately portray the percentage of students exempted."

Standardized Testing

A test becomes a standardized test when it is administered (directions) to all students under the same conditions (time limit) and scored in the same manner for all students. The major purpose of a standardized test is to allow reliable and valid comparisons to be made among students taking the test. Two major types of standardized tests are currently in use; norm-referenced and criterion-referenced.

Norm-referenced Test

A norm-referenced test is a test that has been given to representative samples of students such that norms of performance are established. Each student taking the test receives a score that can be compared to the norm or sample of students. Scores are reported in percentiles or quartiles and/or grade levels. The main purpose of a norm-referenced test is to rank students along a distribution of performance. Thus, norm-referenced tests are likely to have items that are very difficult for the grade level so students can be ranked.

Criterion-referenced Test

Criterion-referenced tests may look much like norm-referenced in that multiple choice items are predominantly used and directions are standardized. The development of criterion-referenced tests is based upon the content all students are expected to learn. Thus, theoretically, every child who has received adequate instruction should be able to answer each of the questions. Scores are based upon the amount of content learned by the student and compared to a 'passing score', not compared to other students. Often scores are reported in achievement level category descriptors such as novice, accomplished, and expert.

What Standardized Tests Can Do

Standardized tests can be used to show areas of student achievement. When used appropriately and keyed to standards and curriculum they can show performance relative to learning goals.

Standardized tests can show program effectiveness. For example, standardized tests can show educators whether the mathematics program is effectively teaching the content covered by the test.

Standardized tests can also show overall school progress toward improving student achievement.

Standardized tests can show gaps in achievement among different groups of students in the school when the data are disaggregated by group.

What Standardized Tests Cannot Do

Standardized tests cannot provide reliable, thorough diagnostic information on each child's needs.

Tests alone cannot improve education without attention and resources to improving instruction, teacher quality, facilities, and access for all students. In fact, over-reliance on standardized testing is potentially detrimental to education and school improvement by transferring resources from instructional programs into the testing program.

Increased standardized testing can narrow the focus of the curriculum to the content that is measured.

Standardized tests can only measure a sample of what is learned in school. Increasing the use and the consequences of testing can create a stressful and anxious learning environment for educators and students, rather than schools where teachers and students view learning as a joyful, inquisitive process that provides a sense of true accomplishment for teachers and students.

Standardized tests, particularly tests with only multiple choice questions, often have difficulty in measuring higher level thinking, deep content knowledge and the ability to construct meaning, solve problems, and analyze critically. Recently states have begun to develop criterion-referenced tests, which are standardized, but are more closely aligned to state content standards than most norm-referenced tests. Innovative item development that has included more open-ended questions has also improved the capacity of tests for measuring different levels of learning and promoting higher-level thinking skills. However, these tests, when constructed correctly, take even more time and expense to administer and score.

Illustrative Language for an RFP To Build Tests To Support Instruction and Accountability

Prepared By Members of the
Commission on Instructionally Supportive Assessment

Convened By

American Association of School Administrators
National Association of Elementary School Principals
National Association of Secondary School Principals
National Education Association
National Middle School Association

October 2001

This report was prepared by the following members of the Commission on Instructionally Supportive Assessment. Commission members' affiliations do not denote institutional endorsement of this document.

David C. Berliner
Regents' Professor of Education
College of Education
Arizona State University

Carol Camp Yeakey
Professor of Urban Politics and Policy
Curry School of Education
University of Virginia

James W. Pellegrino Distinguished Professor of Cognitive Psychology
Distinguished Professor of Education
University of Illinois at Chicago

W. James Popham (Chair)
Professor Emeritus
Graduate School of Education and Information Studies
University of California, Los Angeles

Rachel F. Quenemoen
Senior Fellow for Technical Assistance and Research
National Center on Educational Outcomes
University of Minnesota

Flora V. Rodríguez-Brown
Professor of Curriculum and Instruction/Reading, Writing and Literacy
University of Illinois at Chicago

Paul D. Sandifer (Ret.)
Consultant to the Office of Assessment
South Carolina Department of Education

Stephen G. Sireci
Associate Professor
School of Education
Center for Educational Assessment
University of Massachusetts, Amherst

Martha L. Thurlow
Senior Research Associate, Department of Educational Psychology
College of Education and Human Development
Director, National Center on Educational Achievement
University of Minnesota

Introduction

In a separate report, Building Tests To Support Accountability and Instruction: A Guide for Policymakers¹ the Commission on Instructionally Supportive Assessment identifies nine requirements that must be satisfied if statewide tests are to support both instruction and accountability:

State policymakers might adopt various approaches to address these requirements; for example, by directing a state agency to do so. Many states, however, will likely seek the assistance of external contractors to undertake work related to one or more of the Commission's requirements. Typically, such support is solicited in the form of a request for proposals (RFP) issued to suitable agencies or individuals.

This report presents illustrative language that states might incorporate into RFPs they issue to satisfy the Commission's requirements. These RFPs may be addressed to test contractors, curriculum and instruction consultants, independent researchers, or other agencies depending on the nature of the requirement.

State personnel who write RFPs are encouraged to adapt the illustrative language; hence the report is available in html as well as PDF format. The Commission does not endorse particular procedures for satisfying its nine requirements, nor does it recommend the particular RFP language found in this report. Instead, the language is provided to assist states as they develop RFPs.

The illustrative language in this report focuses on one aspect of the RFPs that states might issue procedures to create, install, and evaluate assessments that meet both instructional and accountability purposes. Other than the tests themselves, states will not need to depart substantially from what is routinely included in RFPs that solicit bids for developing statewide tests (e.g., qualifications of the bidders, reporting requirements, payment schedules). Nor will states need to alter substantially many of the technical issues normally addressed in assessment-related RFPs (e.g., year-to-year equating of tests administered at certain grade-levels, establishing performance standards based on specific standard-setting procedures).

However, as a state moves toward assessments that fulfill both instructional and accountability purposes, and that are fully inclusive, some technical requirements will change to meet the expectation that all students are included in the assessments, and that reports of students' performance support instructional improvement. These changes are not addressed in this report; therefore, states will need to work with their contractors and stakeholders to ensure that the technical requirements of their RFPs are consistent with the purpose and function of the assessments described here.

Finally, before turning to illustrative RFP-language related to each of the Commission's nine requirements, states may wish to consider including introductory language in their RFP such as the following: "The procedures specified in this RFP represent an approach to accomplishing the required work. Bidders are invited, however, to propose alternative procedures provides those alternatives are cost-effective and supply the required services and deliverables."

Illustrative RFP Language

States may solicit proposals from a variety of contractors to conduct work related to the Commission's nine requirements. The illustrative language provided for Requirements 1, 2, 3, 4, and 6 is appropriate for soliciting proposals from test developers. Work related to Requirement 1 also might be solicited from a vendor other than a testing firm. Requirements 5, 7, 8, and 9 specify data collection and evaluation tasks best completed by contractors other than testing firms.

Because Requirements 1, 2, 6 and 7 entail procedures with which architects of state-issued RFPs may be unfamiliar, detailed language is offered in this document. Less detailed language is suggested for Requirements 3, 4, 5, 8, and 9. And in some instances, such as Requirement 8 that deals with professional development programs for educators, the report offers limited language. Experienced staff developers in most states can provide expert assistance in that aspect of the RFP's preparation.

Throughout the report segments of the illustrative language are bracketed to remind readers to modify this text for specific state contexts. For instance, the illustrative RFP-language related to an item-review committee reads: "An item-review committee of [20] members will be assembled at [a site to be recommended by the bidder]." The size of the item-review committee and the site of its meeting are the decision of the individuals preparing the RFP.

In addition, the report includes examples of products or procedures that are described in the RFP language. These examples are presented in italicized text.

Requirement 1

A state's content standards must be prioritized to support effective instruction and assessment.

Illustrative RFP-Language
The tests that bidders develop in response to this RFP must accurately estimate the degree to which a student has attained state content standards² that represent essential or high-priority skills and/or knowledge.

Therefore, bidders will need to carry out a defensible procedure for prioritizing the state's content standards in each of the subject areas/grade levels for which a state test is being developed. This procedure must be based primarily on preferences registered by committees of educators approved by the Department of Education (DOE). ³ Classroom teachers and curriculum specialists must be involved in prioritizing the content standards. Neither (DOE) staff nor the bidder's staff should take a substantive role in prioritizing content standards. It would, however, be appropriate for them to comment on any technical or practical difficulties likely to be encountered in assessing a given content standard.

Because prioritizing will reduce the number of content standards that will be assessed by state tests, it is important for each content standard assessed by the tests to represent an essential or significant skill and/or body of knowledge. It is unlikely that any of the content standards assessed by the test would deal exclusively with knowledge. Rather, it seems more likely that bodies of knowledge would be subsumed under content standards that focus on high-level cognitive skills.

Therefore, it may be necessary to reconfigure some current content standards so that they subsume other content standards. If reconfigurations of content standards result from prioritizing and are proposed, the bidder must provide a justification. In particular, bidders must provide evidence that any content standards subsumed under existing, or reconfigured, standards represent enabling sub-skills or knowledge for the essential skill/knowledge sought.

A useful illustration is represented by assessments of students' written composition skills in which students generate original writing samples. Written communication skill is a significant cognitive competency that embraces a number of sub skills (e.g., written mechanics, content organization) as well as knowledge of the topic addressed in the composition. But these enabling skills are subsumed by a single, powerful content standard. These are the kinds of high-import content standards that should be assessed on the state test(s).

To clarify the DOE's intent, a prioritizing procedure is described below. Bidders are free to modify this process or to use another procedure.

Example of A Procedure to Prioritize State Content Standards

The bidder will convene a panel [20-25 members], [identified in collaboration with DOE] to prioritize the state's content standard in [the subject areas/grade levels]. The panel [Curricular Prioritizing Panel (CPP)] shall be composed of equal numbers of classroom teachers, other educators, curriculum specialists from school districts, and representatives from institutions of higher education who specialize in the content area for which the test is being developed. Typically, different CPPs will be needed for tests in different content areas, although the same panel might prioritize content standards at different grade levels in the same content area.

The bidder will moderate at least one meeting among panel members. [DOE personnel will be present as observers.] Prior to the CPP meeting, the bidder will provide panel members with a set of the appropriate state content standards, a description of the procedures panel members will follow to prioritize the standards during the CPP meeting, and information on required preparation for the meeting. As important, bidders must tell CPP members about all the steps in the prioritization process so members understand how their ratings and rankings of content standards will be used to arrive at final sets of standards the bidder will provide DOE.

Each CPP member will be asked to review the content standards before the meeting. Panel members should be supplied with forms to rate the importance of each standard. The nature of this review will depend on the specific prioritizing process proposed by the bidder. To illustrate, one possible procedure would require panel members to rate each content standard using a rating system such as the following:

1. essential (i.e., standards that are the most important for students to attain);
2. very desirable, (i.e., standards that are extremely important for students to attain);
3. desirable (i.e., standards that are somewhat important for students to attain;
4. optional (i.e., standards that are least important for students to attain).

The bidder must describe the specific language and response categories they intend to use in securing per-standard ratings.

Bidders will compile CPP members' initial ratings into four groups (essential, very desirable, desirable, and optional) so that panel members can review and discuss their initial ratings at the panel meeting. The bidder must indicate decision-rules it employs to place the standards into these categories.

During the CPP meeting panel members will be asked to develop (1) a set of essential content standards ranked from most-to-least important and (2) a set of very desirable content standards ranked from most-to-least important. To expedite this process, the bidder could present panel members with the content standards they have rated individually as essential and very desirable, ask them to rank each set of standards from most to least important, discuss, and as needed, revise their ratings. The bidder may suggest more than one iteration of this process. During the meeting, panel members also might be asked to reformulate existing content standards to represent broad, significant outcomes. If DOE prefers, it could carry out this reformulation and provide these reconfigured standards to panel members for their consideration.

As a result of the CPP members' work, the bidder will provide DOE (1) a set of essential content standards ranked from most-to-least important and (2) a set of very desirable content standards ranked from most-to-least important.

Requirement 2

A state's high-priority content standards must be clearly and thoroughly described so that the knowledge and skills students need to demonstrate competence are evident.

Illustrative RFP-Language
So that educators understand clearly the skills and knowledge in the content standards that are assessed by state tests developed in response to this RFP, bidders must supply a short, clear, and concise assessment description for each standard. These assessment descriptions, which should not exceed three paragraphs, must describe the cognitive domains and cognitive demand(s) students are expected to meet. In addition, each assessment description must be accompanied by a minimum of [three] different illustrative item-types that might be used to assess the standard. [These sample items must be labeled as "Illustrative Item-Types."]

Each assessment description should focus on the nature of the cognitive demand(s) to which students are expected to respond. Educators who read the assessment description, and the content standard on which it is based, should be able to arrive at an accurate understanding of the intellectual operation(s) that the content standard requires of students. Moreover, if any enabling sub skills or bodies of knowledge are identified in the assessment description, educators will understand the nature and importance of these sub skills or bodies of knowledge.

Bidders must identify the kinds of personnel who will develop the assessment descriptions. The DOE strongly urges bidders to involve both instructional and measurement staff. In addition, educators, especially classroom teachers, must be involved in the generation and/or review of the assessment descriptions since they will be the important users of the descriptions.

All assessment descriptions should be pilot-tested with small groups of educators, and bidders should describe acceptable outcomes from these pilot tests as well as any decision-rules they will employ to determine a description's adequacy. Bidders also should indicate the kinds and numbers of educators who participate in the pilot-tests.

To assist bidders, an illustrative assessment description that shows one way of satisfying this section of the RFP is provided below.

Example of an Assessment Description

Overview: This assessment description and the illustrative tasks that follow it are intended for an eleventh- and twelfth-grade U.S. history course. If the skill in this assessment description is promoted at earlier grade levels in which U.S. history is taught, the language and cognitive complexity of the illustrative items will need to be simplified.

To measure this skill, the state must identify (and communicate to educators) eligible historical events for students to consider. As an example, the following historical events were identified as suitable by one district for its eleventh- and twelfth-grade U.S. history courses: Constitution, Territorial Expansion, Civil War, Reconstruction, Industrial Revolution, Imperialism, World War I, Depression, New Deal, World War II, Cold War, Civil Rights, Viet Nam, Communication Revolution.

Content Standard: Students will be able to draw upon historical lessons to deal with today's societal problems.

Assessment Description: Given a prose account of a real or fictitious current problem, as well as a proposed solution to that problem, students will be able to respond appropriately to any one or any combination of the following subtasks:

Subtask 1: Event Selection. Identify at least one significant historical event (such as the industrial revolution) that is, at least in part, germane to the problem and its proposed solution.

Subtask 2: Event Justification. Justify the relevance of the identified historical event(s) to the problem and its proposed solution.

Subtask 3: History-based Prediction. Make a defensible history-based prediction regarding the proposed solution's likely consequences.

Subtask 4: Defense of a Prediction. Support that prediction on the basis of parallels between the identified historical event(s) and the proposed problem-solution.

Students will be presented, orally or in writing, with a real or fictitious current day problem-situation along with a proposed solution to that problem. (See illustrative items below.) After they have had an opportunity to consider the problem and the proposed solution, they will be asked to supply written or oral responses to one or more of the four subtasks listed above.

Although they may be asked to supply responses to individual subtasks so that their mastery of those particular types of subtasks can be determined, all students will ultimately be required to respond to a comprehensive task such as the one illustrated below. When they respond to a comprehensive task, of course, they will be given a problem and proposed solution that are different from the ones used for individual subtasks.

Illustrative Tasks/Items. Directions: Read the fictitious problem described below as well as the proposed solution to that problem, then respond to each of the tasks indicated. (Note: These illustrative items would not be presented on the same test form.)

WAR OR PEACE

Nation A, a large, industrialized country whose population is almost 100,000,000, has ample resources, and is democratically governed. It also owns two groups of islands that, although distant, are rich in iron ore and petroleum.

Nation B, a country with far fewer natural resources and a population of only 40,000,000, is about one-third as large as Nation A. Although much less industrialized than Nation A, Nation B is as technologically advanced as Nation A. A three-member council of generals governs Nation B.

Recently, without any advance warning, Nation B ruthlessly attacked Nation A. As a consequence of this attack, more than half of Nation A's military equipment was destroyed. After its highly successful surprise attack, Nation B's rulers have proposed a "peace agreement" calling for Nation A to turn over its two groups of islands to Nation B. If Nation A does not concede the islands, Nation B's rulers have threatened all-out war.

Nation A's elected leaders are fearful of the consequences of the threatened war because their military equipment is now much weaker than that of Nation B. Nation A's leaders are faced with a choice between (1) peace obtained by giving up the islands or (2) war with a militarily stronger nation.

Nation A's leaders decide to declare that a state of war exists with Nation B. They believe that even though Nation B is now stronger, in the long term Nation A will prevail because of its greater industrial capability and richer natural resources.

1. Nations: U.S. and Italy I; Conflict: World War I
2. Nations: U.S. and North Korea; Conflict: Korean "Police Action"
3. Nations: U.S. and Spain; Conflict: Spanish American War
4. Nations: U.S. and Japan; Conflict: World War II

Requirement 3

The results of a state's assessment of high-priority content standards should be reported standard-by-standard for each student, school, and district.

Illustrative RFP-Language
Tasks that the successful bidder completes under Requirement 1 will result in a list of prioritized content standards. To satisfy this section of the RFP, bidders must provide DOE with strongly supported recommendations regarding which of the highest-priority content standards (see Requirement 2) will be assessed during the time available for testing [which the DOE addresses elsewhere in this RFP].

In addition, the tests that are developed in response to this RFP must enable the DOE and educators to make valid inferences about students' performance related to each content standard. Therefore, bidders must (a) specify the number of assessment items/tasks related to each standard, and (b) provide evidence that the tests they construct will yield accurate per-standard evidence of a student's performance.

Bidders also must provide the DOE with information on the probable accuracy of reporting test results standard-by-standard at various levels of aggregation. That is, bidders must supply estimates of the likely accuracy of per-standard reports of student performance at the state, district, school, and student levels. Similarly, bidders must provide probable accuracy-estimates at different reporting levels not only for total-group results, but also for disaggregated student groups (e.g., racial, disability, limited English proficiency, and gender subgroups). To ensure greater accuracy, bidders may choose to report results aggregated across standards in addition to per-standard reporting. If this is the case, the proposed procedures for such aggregation must be described.

At the same time, bidders must describe how per-standard test results will be reported to relevant constituencies, for example, to educators, parents, and students.

The DOE recognizes the technical problems associated with providing accurate estimates of individual student's performance on each content standard. Yet, it is our conviction that per-standard results, even results whose accuracy in certain instances will be somewhat reduced, will prove to be instructionally useful to educators who currently have no evidence about students' performance related to individual content standards.

Requirement 4

The state must provide educators with optional classroom assessment procedures that can measure students' progress in attaining content standards state tests do not assess.

Illustrative RFP-Language
As a consequence of prioritizing the state's content standards, many important standards will not be assessed by state tests that will be developed in response to this RFP. Nevertheless, these standards should be assessed. The DOE believes this can be done via classroom assessments that can either be developed by, or supplied to, educators. To this end, the DOE intends to make available optional classroom assessments the state's educators can use if they choose.

Therefore, bidders must describe how they would create assessment descriptions and build optional classroom assessments for the content standards that are not assessed on state tests. Each bidder's proposal must indicate awareness of the need to create (a) valid classroom assessments that busy educators are likely to use; and (b) reliable classroom assessments whose results schools and districts can report to the DOE for accountability purposes.

Bidders must describe the procedure they will use to choose the content standards designated for the generation of optional classroom assessments. DOE must approve the content standards finally selected for this activity.

For each content standard chosen, and at each grade level for which an optional classroom assessment is to be produced, bidders must create two forms of each assessment. This will provide educators more flexibility in how they use the assessments. For example, the assessments could be employed as instructional pretests and posttests. [Although these test do not need to be of equal difficulty psychometrically, bidders must create test forms that are reasonably similar in difficulty levels.]

Bidders must describe procedures they will employ to assess and provide the DOE with data on the optional classroom assessments' validity and reliability. To this end, bidders must describe how they have developed, piloted tested the assessments with students and educators, and revised and finalized the assessments. Each test form must be pilot-tested with students and educators. In addition, each test form should be pilot-tested on at least [six] students in a session in which students' reactions to the test are sought and documented after they complete the test form.

Because the costs associated with this activity will vary based on the number of tests produced, bidders should supply cost estimates for the assessments in [10-test] increments [(two forms per test)], that is, for [10] pairs of optional classroom assessments, [20] pairs of classroom assessments, etc.

The bidder must provide the DOE with recommendations for disseminating the optional classroom assessments to educators through online and print avenues. Any information educators need to administer, interpret, and/or score the assessments must be made available to the DOE at the time the classroom assessments are delivered. The assessments must be easy to use and easy to score.

Requirement 5

A state must monitor the breadth of the curriculum to ensure that instructional attention is given to all content standards and subject areas, including those that are not assessed by state tests.

Illustrative RFP-Language
State tests focused on high-priority content standards could inadvertently narrow curriculum coverage. Therefore, bidders must propose cost-effective methods that they will use to provide annual estimates of curricular coverage at the state, district, and school levels. These methods may reflect either qualitative or quantitative procedures, or a combination of both.

Bidders' proposed methods of monitoring curricular breadth should provide evidence regarding both the intended curriculum and the enacted curriculum. In other words, bidders must describe how they propose to monitor curricular aims as well as the curriculum that is actually implemented, which may or may not correspond closely with the intended curriculum. Bidders must make clear how their methods of monitoring curricular breadth would address this possibility.

In addition, bidders must propose a reasoned approach for reporting, and including in accountability processes, the results of local classroom assessments measuring standards that are not part of the state test (see Requirement 4). These local assessments must be an essential component of the monitoring of statewide curricular breadth.

Requirement 6

A state must ensure that all students have the opportunity to demonstrate their achievement of state standards; consequently, it must provide well-designed assessments appropriate for a broad range of students, with accommodations and alternate methods of assessment available for students who need them.

Illustrative RFP-Language
All students must have the opportunity to demonstrate their achievement of the same content standards. Therefore, to satisfy this section of the RFP, bidders must design state tests that allow the maximum number of students possible (and students with diverse characteristics) to take the same assessments without threat to the validity and comparability of the scores.

To this end, bidders must demonstrate how they will develop "universally designed assessments." Designed from the beginning to allow participation of the widest range of students, these assessments result in valid inferences about the performance of all students, including students with disabilities, students with limited English proficiency, and students with other special needs.

While universally designed assessments will not eliminate the need for all accommodations (e.g., tests in Braille or large print, extended time to take the test, individual test administration), they can significantly reduce the need for them. As important, universally designed assessments increase the variety of accommodations that can be used without threat to the validity and comparability of the scores. Overall, these assessments result in inclusive accountability measurement, and they provide instructionally supportive information across the full range of students.

Universal design processes do not change the definition of the construct measured by the assessment but focus instead on the following assessment-design characteristics that bidders should integrate into their test development procedures.

Test conceptualization. Bidders must define the construct(s) to be measured precisely and explicitly so tests measure the construct while minimizing the effects of irrelevant factors. Bidders also must include the full range of students in the definition of the target population that will take the assessments.

Test construction. Bidders must develop items that minimize the effects of extraneous factors (e.g., avoid unnecessary use of graphics that cannot be presented in Braille, use font size and white space appropriate for clarity and focus, avoid unnecessary linguistic complexity when it is not being assessed). Bidders also must provide for a full range of test performance to avoid ceiling or floor effects, and they must develop an item pool of sufficient size to permit the elimination of items that are not found to be universally appropriate during the test tryout and item analysis.

Test tryout, analysis, and revision. Bidders must include a full range of students in the tryout sample (e.g., students with disabilities, students with limited English proficiency, other students with special needs). Because there may be constraints in sampling due to the low numbers of students with specific characteristics, bidders may need to identify over-sampling strategies, (e.g., select groups of items for which additional sampling will occur). Bidders should include the use of accommodations during the test tryout.

As part of required test item analysis, bidders must analyze item characteristics to determine which items can be used with the full range of students and with accommodations. This includes examining items for evidence of disability bias and eliminating such items during test revision. Bidders must (a) include the full range of students and (b) use accommodations in any test administration conducted during tryout and revision.

Identification of other assessment methods. Bidders must describe how they will assist educators in devising alternative methods of assessing student progress (e.g., panel review, performance or portfolio assessment) for students who cannot take the universally designed state tests. Alternative methods must assess progress toward the same content standards and the same levels of performance that are assessed in the state tests. This requirement is in addition to any responsibility bidders must assume for developing federally required alternate assessments.

Requirement 7

A state must generally allow test developers a minimum of three years to produce statewide tests that satisfy Standards for Educational and Psychological Testing and similar test-quality guidelines.

Illustrative RFP-Language
This RFP provides sufficient time (i.e., three years for a contractor to create new state tests) for bidders to meet the assessment profession's standards related to test construction. [The major timeline milestones set forth elsewhere in the RFP indicate the major deadlines with which bidders must comply.]

As described elsewhere in this RFP, bidders must describe how they intend to use the time provided to create state tests that satisfy the DOE's need for accountability evidence and also support educators' instructional efforts (see Requirements 2, 3, and 4). The DOE regards the precepts embodied in the Standards for Educational and Psychological Testing⁴as key to ensuring that state tests will be suitable for these purposes. Therefore, bidders must explain how they will comply with the Standards' stipulations (e.g., assembling compelling validity evidence).

Bidders also must describe any evaluative activities in which they will engage to ensure that the state tests and the optional classroom assessments (see Requirement 4) they produce, and any related deliverables or services, are of high quality. These evaluative activities might involve the use of "cognitive laboratories" in which small groups of students respond to test items and describe to members of a bidder's staff the nature of the intellectual operations that they have employed to complete the items. Bidders who choose to use such evaluative activities must describe the decision-rules they will employ to revise test items using evidence they have obtained from the activities.

If bidders plan to employ judgmental reviews of test items, the key elements of those reviews must be explicit. At a minimum, these reviews must address the item-quality issues embodied in the following four questions:

Standards congruence: Will students' responses to a test item help educators accurately determine whether students have demonstrated the knowledge and/or skill embodied in the designated content standard the item is intended to assess?

Out-of-school factors: Is the test item free of content that would make a student's likelihood of answering it correctly be dominantly influenced by factors other than what has been taught in school?

Instructional sensitivity: if a teacher has provided effective instruction related to this item's content standard, it is likely that most of the teacher's students will answer the item correctly?

Absence of bias: Is a test item essentially free of content that might offend or unfairly penalize students because of personal characteristics such as race, gender, religion, primary language, or disability?

Bidders who plan to use judgmental reviews of test items must describe the key elements of those reviews. To clarify the DOE's intent, a review process is described below. Bidders are free to modify this process or use another procedure.

Example of a Process to Review Assessment Descriptions and Test Items

Because well-formed assessment descriptions will play a pivotal role in the subsequent review of potential test items, it is important to refine all assessment descriptions as soon as possible. Therefore, the bidder will convene committees of experienced educators to review both the assessment descriptions (by mail) and the pool of proposed test items (in person).] These committees, one for each content area and grade level for which a test is being constructed and consisting of [20-30] members, can be described as [Materials Review Committees (MRCs)]. DOE personnel will designate the membership of each MRC, but most members will be classroom teachers or those who work closely with classroom teachers. The bidder will be responsible for recruiting members for the MRCs, although DOE will supply a list of potential invitees [plus suitable cover letters inviting participation in an MRC]. The bidder also will prepare an invitational letter for potential MRC participants. That letter, approved by DOE, will describe the state tests the bidder is developing and how the MRC's review of assessment descriptions and test items fits into test development.

As a first step in the review, the bidder will provide each MRC member with the prioritized content standards that the state test is designed to measure and will draft assessment descriptions for each standard. Each content standard and assessment description is intended to provide educators with clear descriptions of the cognitive skills assessed by the state test.

Bidders will ask reviewers to (a) read each content standard and its accompanying draft assessment description and (b) evaluate each description's usefulness in instructional planning and ease of use. Reviewers' judgments should be made in response to the following questions or to improved versions of these questions suggested by the bidder. [Abbreviated versions of these questions also could be provided to reviewers as they rate assessment descriptions.]

Usefulness for instructional planning: 1. After reading the content standard on which this assessment description is based, and the description itself, would educators have a sufficiently clear idea of the cognitive demands required of students to plan effective instruction for students related to the content being measured? (Reviewers would respond: definitely yes, probably yes, probably not, or definitely not.) 2. If you have suggestions for improving this assessment description's usefulness for instructional planning, please supply them below.

Ease of Use: Is the assessment description sufficiently understandable and concise so that most educators would find it easy to read and use? (Reviewers would respond: definitely yes, probably yes, probably not, or definitely not.) 2. If you have suggestions for making the assessment more useful to educators, please supply them below.

Bidders will supply MRC members with pre-stamped envelopes to return their reactions to the draft assessment descriptions. After those reactions have been considered, it is likely that some of the descriptions will need to be modified, thereby leading to another round of by-mail MRC reactions to any modified assessment descriptions. Appropriate DOE staff must approve all modifications. It is imperative that when the MRCs meet to review test items, they have assessment descriptions to work with that the vast majority of MRC members have judged to be easy for educators to use and useful in instructional planning.

As a second step in the review, bidders will convene members of the MRC to review test items under conditions that are carefully monitored by members of the bidder's staff. The bidder must describe in detail the security procedures it will employ during these item-review sessions. Moreover, the main elements of the item-review procedure should be described when bidders respond to this RFP. For example, the nature of any planned orientation the bidder will provide MRC members must be well described.

The bidder may choose to use an item-review process in which reviewers:

1. Individually answer specified questions for each test item in a subset of items.
2. As a group consider the subset of items to see if any need to be modified.
3. Individually examine any test items that have been altered.
4. As a group reconsider any altered items.

Bidders may propose other item-review procedures, which they must describe in detail. In addition to the test items, reviewers need the final assessment description for each content standard as well as the content standard. The review process suggested here is particularly useful if MRCs review items in subsets of [10-15] items. The individual/ group procedure continues until all items have been reviewed.

The item-review questions the MRC members answer must address, at least, the item-quality issues embodied in the following four questions:

Standards congruence: Will students' responses to a test item help educators accurately determine whether students have demonstrated the knowledge and/or skill embodied in the designated content standard the item is intended to assess?

Out-of-school factors: Is the test item free of content that would make a student's likelihood of answering it correctly dominantly influenced by factors other than what has been taught in school?

Instructional sensitivity: If a teacher has provided effective instruction related to this item's content standard, is it likely that most of the teacher's students will answer the item correctly?

Absence of bias: Is a test item essentially free of content that might offend or unfairly penalize students because of personal characteristics such as race, gender, religion, primary language, or disability?

Bidders may propose the use of additional item review questions. It should be noted that this suggested review process requires bidders to have indicated (at item-review time) the chief content standard that each item is intended to assess.

The length of time the MRC meets depends on the number of items to be reviewed. Based on experience in conducting item-review meetings, bidders should propose and defend the probable length of such meetings. [Bidders will be responsible for all travel-related expenses associated with the MRC meetings and for any stipends paid MRC members.]

At the conclusion of each MRC session, the bidder will summarize the review data on all evaluative dimensions employed (e.g., for each of four questions cited above). Then the bidder and the DOE will designate test items deemed suitable for field-testing. The vast majority of the items selected for field-testing will have received positive evaluations on all evaluative dimensions employed during the MRC sessions.

Bidders are to describe, in sufficient detail, the procedures they intend to employ to ensure item security. For example, if particular types of item-monitoring procedures are to be used in order to maintain security of the item pool, those procedures should be explicated in a bidder's proposal. Any type of confidentiality form that bidders will ask MRC members to sign should also be described.

Requirement 8

A state must ensure that educators receive professional development focused on how to optimize children's learning based on the results of instructionally supportive assessments.

Illustrative RFP-Language
This requirement, which ties professional development to the state's testing program, is intended to remind bidders that test information needs to be acted on, not left simply as a report to schools, school districts, and the state. The DOE intends for professional development to be an integral part of the state's testing program so that assessment results inform instructional improvement.

Suitable professional development involves educators in the design, implementation, and continuous improvement of a variety of strategies that result in building knowledge and skills to improve instruction in schools and classrooms. Therefore, the DOE anticipates development of an assessment-related professional development system that includes components such as: [e.g., linkages among practitioners within content or subject areas; best practice networks; collaborative analyses by educators of exemplary videotaped lessons; availability of training and support on a variety of effective instructional and assessment strategies.]

Although it will be the DOE's responsibility to coordinate these assessment-related professional development activities, the DOE fully expects bidders to cooperate in its conceptualization of these professional development activities. Moreover, the DOE expects the contractor to provide assessment-relevant insights about how to make the professional development program most effective.

Therefore, bidders must describe the kinds of contributions dealing with assessment and/or instruction that they plan to make to the development and, possibly, delivery of professional development activities to support the state and optional classroom assessments they are developing in response to this RFP.

Among the bidders' suggestions should be a demonstration of how educators can use disaggregated student performance data to design differentiated instructional strategies so that all subgroups of students are achieving at high levels. For example, bidders may demonstrate how reports of assessment results for districts, schools, and individual students can be used in instructional planning. Similarly, bidders may describe how they will develop a training package to assist educators in using assessment data for instructional decision making.

Requirement 9

A state should secure evidence that supports the ongoing improvement of its state assessments to ensure those assessments are (a) appropriate for the accountability purposes for which they are used, (b) appropriate for determining whether students have attained state standards, (c) appropriate for enhancing instruction, and (d) not the cause of negative consequences.

Illustrative RFP-Language
Because the requirements in this RFP specify state tests and optional classroom assessments that will change the nature of the state's assessment program, the DOE is interested in receiving bids from contractors to evaluate the assessment program.

Therefore, successful bidders must function independently from any test contractor(s) involved in the development of state tests and optional classroom assessments, and in the implementation of the assessment system. In addition, these independent evaluators must have expertise and experience in designing and conducting program evaluations, reporting results, and using results for program improvement. Bidders must describe the formative and summative evaluation procedures they will use to identify strategies for continually improving the state's assessment system.

The evaluative procedures bidders employ must provide the DOE with information about whether state assessments are:

1. appropriate for the accountability purposes for which they are used;
2. appropriate for determining students' attainment of the state's content standards;
3. sensitive to instructional quality; and
4. not the cause of negative consequences for students, educators, schools, and school districts.

To satisfy this section of the RFP, bidders must include in their proposals:

1. initial evaluation questions linked to the primary four areas of interest above;
2. initial data gathering strategies, including qualitative and quantitative strategies as appropriate;
3. initial analysis strategies in each of the primary four areas of interest, including a description of stakeholder involvement as meaning is derived from the data; and
4. proposed interim and final report formats and strategies for building consensus for recommendations to change the system.

In addition, because the evaluation activities will be conducted in collaboration with the DOE, bidders must describe the nature of any expectations regarding the DOE's participation in evaluation design and implementation.

A Final Reminder

This report is designed to assist states that may choose to issue RFPs that solicit contractors to assist in the development of state tests and optional classroom systems advocated by the Commission on Instructionally Supportive Assessment. The RFP-language provided here illustrates ways in which an RFP might be worded. It should not be concluded that the Commission's members, either individually or collectively, are recommending one way for states to proceed.

_______________________

¹Building Tests To Support Accountability and Instruction: A Guide for Policymakers, is available online from the associations that convened the Commission on Instructionally Supportive Assessment.

²States will need to make sure that any illustrative language in this report meshes with the manner in which their content standards have been developed and approved.

³Department of Education (DOE) is used throughout the report to designate the state agency responsible for issuing RFPs. Some states use different designations for this agency, or issue RFPs through another state agency.

Other Resources

The National Assessment Governing Board (NAGB) is an independent, bipartisan group whose members include governors, state legislators, local and state school officials, educators, business representatives, and members of the general public. Congress created the 26-member governing board in 1988 to set policy for the National Assessment of Educational Progress (NAEP), commonly known as the "Nation's Report Card."

The Education Testing Service, the nation's largest testing service, administers the SAT, Advance Placement and PRAXIS tests for beginning teachers. It also conducts research into a variety of standards and assessment issues.

The Education Commission of the States' (ECS)  online database gives you a "real-time" snapshot of how your state is doing in meeting goals of the No Child Left Behind Act. ECS also offers background and information on a host of education issues, including accountability and teacher quality.

The National Conference of State Legislatures provides background on state accountability actions over the past three years.

Testing

"What Tests Can and Cannot Tell Us," the October 2002 edition of Test Talk for Leaders from the Center on Education Policy, explains the strengths and limitations of testing.

The National Center for Fair & Open Testing (FairTest) works to end  abuses, misuses and flaws of standardized testing and to make certain that evaluation of students and workers is fair, open and educationally sound. FairTest’s Web site includes a wide range of resources on K-12 and university testing for teachers, administrators, students, parents and researchers.

The National Board on Educational Testing and Public Policy at Boston College recently asked thousands of teachers from across the nation how they think high-stakes tests are affecting teaching and learning. The majority reported that there is so much pressure for high scores on state-mandated tests that they have little time to teach anything not covered on the test.

The Consortium for Policy Research in Education's Web site contains a database with a detailed description of each state's testing and accountability programs.

The Center for Research and Evaluation on Student Standards and Testing is a federally funded center for research and information on student testing and accountability.

Teacher Quality

The National Commission on Teaching and America's Future (NCTAF) has done a number of studies on the relationship between teacher quality and student achievement. The landmark report, What Matters Most: Teaching for America's Future (1996) (, PDF, 46 pp.), calls for teacher licensure based on testing and demonstration, reinventing preparation and professional development programs, and organizing schools to focus on teacher quality and student achievement.

NCTAF also released Solving the Dilemmas of Teacher Supply, Demand, and Standards: How We Can Ensure a Competent, Caring, and Qualified Teacher for Every Child (2000) ( PDF, 43 pp.), which calls for raising teacher standards and equalizing salaries between lower and upper income districts, establishing licensure reciprocity agreements among states, providing incentives for National Board for Professional Teaching Standards certification, and providing scholarships in teacher shortage fields.

The National Board for Professional Teaching Standards is an independent board that sets rigorous standards for what teachers should know and be able to do and offers voluntary national certification for accomplished teachers.

The Education Testing Service's (ETS) study  Academic Quality of Prospective Teachers: The Impact of Admissions and Licensure Testing (May 1999) ( PDF, 46 pp.) found that teachers in academic subject areas have academic skills that are equal to or higher than those of the larger college graduate population.

The Relationship Between Standards and Resources

ETS's recent report  An Uneven Start: Indicators of Inequality in School Readiness (March 2002) ( PDF, 77 pp.) found that children start kindergarten with different levels of reading and mathematics skills that are often related to gender, race/ethnicity, socioeconomic level and age.

The Public Education Network is engaged in an initiative in five sites that include communitywide efforts and state-level advocacy to ensure the allocation of resources so that all children have the support they need to achieve at high levels.

Data-Driven Decision Making and School Report Cards

NEA Student Achievement Team

January 2001

What are school report cards?

School report cards are reports issued by a school and/or district that present data and information about the status of the school. School report cards are becoming a major tool in the move to make schools more accountable to the public and policymakers. Supporting assumptions that underlie the use of school report cards suggest they will: inspire low performers to improve, provide for greater accountability, improve public confidence in schools, improve communication with parents and community involvement, and provide greater clarity of school goals. Meanwhile, critics of school report cards respond that: test scores are overemphasized, schools do not fit a one-size fits all model, school comparisons are unfair, and schools with low performance are negatively impacted.
Brown, R.S. 1999. "Creating school accountability reports." (AASAOnline School Administrator has more publications.)

School report cards vary tremendously across the states. A review of a sample of school report cards by the KSA-Plus Communications group found many do not present a clear message and the audience for the reports is often unclear. Ed Week has published a full discussion of these issues.

What kinds of data are included on school report cards?

Chief among the data that are published is student achievement test scores on state standardized tests. Some states mandate that student achievement test scores are the only piece of information that is published. Other states allow local discretion to add data on other school processes that affect student achievement. Types of data that may be applicable to include in school report cards are outcome data and data on the school processes that affect those outcomes. Be leery of including data on non alterable variables such as parent income. Examples of outcome data include student achievement test scores, attendance rates, dropout rates, and promotion and graduation rates. Examples of school processes that affect outcomes include per-pupil expenditures, class size, teacher certification and qualifications, teaching loads, instructional materials, discipline and school safety, school facilities, professional development opportunities, curriculum and instructional practices, school leadership, and special services for students. For a fuller discussion of data on school report cards see Ten Recommendations for Reporting Data to the Public at Ed Week.

Why are school report cards important?

The 2000 edition of Quality Counts from Education Week reports that 45 states are using school/district report cards in their accountability efforts. Currently, only 27 states officially rank schools based upon report cards. Many schools, either through choice or through legislative accountability mandates, have begun publishing school reports in recent years. Due to this prevalence of school report cards and the Bush administration's proposals to make them mandatory for all schools, an Association focus on the development of effective school report cards is appropriate. Local NEA affiliates need to get involved to ensure the information presented is balanced and fair and to make the information useful for ongoing school improvement activities.

How are school report cards and data-driven decision making related?

A focus on school report cards offers an immediate and tangible activity for educators and the local affiliate to become involved in accountability and data driven decision making. In order to produce school report cards, a significant investment must be made at the school level to collect and analyze a large amount of data. Since this process is also the core of data-driven decision making, involvement in school report cards will initiate data-driven decision making for a school. Local affiliates and members can provide guidance on all phases of data collection and analysis in order to make the data and information on the school report card useful for school improvement and advocacy by the local affiliate.

Where can I find examples of school report cards?

Sample school report cards are easily obtainable. The following three sources are suggested.

See Ed Week  (PDF, 2 pages) for a sample report card prepared by a KSA-Plus Communications consulting group.

Check with your district office or your state's department of education web site to see if your state has a prescribed format. Most states are putting district performance reports online with links to individual school data reports.

Enter school report cards into your Internet search engine and a host of school sites from across the country will appear. Surf those results for samples.

Also see Reporting Results: What the Public Wants to Know   (PDF, 20 pages).

How can the local affiliate use school report cards?

Three uses are primary for the local affiliate. School reports cards can be an avenue to promote public support for public education, a vital tool for school improvement, and a source of data and information for advocacy.

The public reporting of school progress can become a means of communication with the community to build public support and confidence. The data in the report can also become information upon which the school and the community can enter into common agenda building activities toward school improvement. The release of data should be accompanied by teacher interpretation of the scores and other supporting information and activities that will engage parents and the community. In order to accomplish this change in perspective, educators should increasingly view themselves in charge of the data and communications about their school.

The data and information collected and analyzed to develop the report cards can be the same information that is used in data-driven decision making, the basis for effective school improvement activities and plans. Collecting and using data from multiple sources using a variety of data collection techniques will allow a more complete picture of a school's strengths and needs. Data on the conditions of teaching and learning and other variables that impact student achievement not only allows a school to make targeted decisions about improvement, but allow the consumers of the report cards to understand the complexities of the schooling process. For a discussion of using school reports cards in school improvement see Gaines, G.F. and L.M. Cornett. 1992. School accountability reports: Lessons learned in SREB states. Atlanta: Southern Regional Educational Board.

Local affiliates can use the data from school report cards in their advocacy on such issues as equalization of resources, staffing, teaching loads, and other areas of concern with boards of education and state and local policymakers. Arguments supported by data are usually more powerful than arguments supported by perceptions (Data-driven Decisionmaking and Association Advocacy.)

In some states, Association affiliates have been able to cooperate with policymakers to agree upon the types of data and information to be included on school report cards. In other states, negative agendas for the report cards have been clear and Association efforts at involvement have been stymied. In the latter cases, some affiliates have decided to move forward with producing their own report cards on schools in order to provide balanced and fair data to combat the negative practice of using only student achievement test scores as a school evaluation method.

Two issues about local affiliate involvement in school report cards should be mentioned. First, educators will be accused of making excuses for any low achievement or weaknesses. Second, if data are collected systematically on each school in a particular district, the data may show that some schools are more effective in terms of student achievement than others with similar characteristics, moving the discussion from school effectiveness to school efficiency. Be aware that school report cards are being used in marketing communities as good places to live, real estate advertisements, and, most recently, in child custody cases.

NEA Helps Develop Assessment of
Student Academic Growth

NEA has partnered with the American Institutes for Research in the development of the federally funded  National Center for Progress Monitoring.

The Center's mission is to provide technical assistance and professional development to states and districts about the use of ongoing curriculum-based assessment to measure and monitor student academic growth -- particularly in reading and mathematics.

One of the Center's first projects is to sponsor a training conference July 22-23 in Washington, D.C. The institute will focus on student progress monitoring for reading.

Because NEA is a partner organization, NEA members are eligible for a special discount registration fee of only $69. NEA members can register online.

Progress Monitoring Helps Target Instruction

Progress monitoring is a scientifically based, but underutilized practice that is used to assess students’ academic performance and evaluate the effectiveness of instruction and instructional interventions.

Used with an individual student or an entire class, progress monitoring results in more efficiently targeted instructional techniques and goals, which together move students to faster attainment of important state standards of achievement.

Last fall, the U.S. Department of Education's Office of Special Education Programs funded a national technical assistance and dissemination center dedicated to the implementation of scientifically based student progress monitoring for grades K-5.

Located at the American Institutes for Research in Washington, D.C., the Center works in conjunction with experts at Vanderbilt University, a National Advisory Committee, a Technical Review Committee, and key association partners, including NEA.

NEA, Other Partners Help Assess, Promote

The Center's National Advisory Committee provides insights into the process for scaling up student progress monitoring. The Technical Review Committee performs an important role in evaluating the scientific rigor of student progress monitoring models and materials to ensure that the information provided by the Center focuses on scientifically based practices. Association Partners help the Center in assessing and promoting the implementation of Progress Monitoring.     

NEA members are invited to frequently visit the Center’s Web site to find answers to frequently asked questions, discover further details about implementing progress monitoring, access related links, and more.     

NEA members also are encouraged to share information with the Center about student progress monitoring practices that they are implementing, as well as useful information and resources. To contact the Center, send e-mail messages to studentprogress@air.org.

NEA Resources

Learn what the Elementary and Secondary Education Act (also called the "No Child Left Behind" Act) says about testing and assessment, as well as what NEA is doing and what you can do about new federal testing requirements.

In Testing Plus, NEA argues that true student assessment must include more than the high-stakes, paper and pencil tests that are all the rage with some politicians.

See the Teaching Experience area of this Web site for these and other articles on assessment:

• "The Mismatch Between Assessment and Grading" looks at alternative assessments in mathematics.
• "Linking Assessment and Teaching" argues that the early years of schooling, prekindergarten through third grade, is the time to begin the process of assessing children's performance related to standards.
• "Analyzing Student Work" shows how measuring students' understanding can help teachers improve their own practice.
• "A Value-Added View of Pupil Performance" has examples for measuring what an individual has learned.
• "Charting the Course of Student Growth" examines what we mean by results.

The NEA report Balanced Assessment: The Key to Accountability and Improved Student Learning (2003) describes how standardized tests can be coordinated with classroom-based assessment to produce a more complete picture of student progress and school quality.

The National Assessment of Educational Progress (NAEP) has taken on a new role for states in educational improvement. Under the ESEA, states must participate in NAEP reading and mathematics assessments at grades 4 and 8 every two years. State participation in NAEP has previously been voluntary. NEA has prepared a list of key questions about NAEP and ESEA.

Building Tests to Support Instruction and Accountability: A Guide for Policymakers: An independent commission of testing experts was convened by NEA and the American Association of Administrators, the National Association of Elementary School Principals, the National Association of Secondary School Principals and the National Middle School Association. The experts were asked to make recommendations for making large-scale assessments useful for both instruction and accountability.

Illustrative Language for an RFP to Build Tests to Support Instruction and Accountability: The independent commission of test experts referenced above took the additional step of translating their nine requirements into language that could be used by states in developing Requests for Proposals to build tests that support instruction and accountability.

ESEA's Testing Provisions: Guidance from an Independent Commission's Requirements  (PDF, 281k, 28 pp.): The independent commission of testing experts offers advice on how to implement the testing provisions of ESEA.

Advancing Standards: A National Call for Mid-Course Corrections and Next Steps: This report asks educators and policy makers to reassess the direction of standards-based education and make corrections to raise achievement and maintain equity.

A Tool for Auditing Standards-Based Education  (MS Word, 31 pp): To facilitate the necessary changes in the direction of standards-based education, NEA developed this tool for state affiliates to assess the status and plan necessary changes in direction for standards-based education.

Multiple Indicators

Multiple indicators are necessary to measure the multiple systems that operate within a school. Measuring student achievement is not the same as measuring the quality of teaching and learning. Multiple indicators are an improvement to using only student achievement scores as the measure of student and school success. The core principle is that better information about education can help improve education.

Multiple indicators necessitate multiple methods of data collection and reporting. Results from measuring multiple indicators yield voluminous data on the outcomes and the operation of a school that is more comparable and reliable. This data allows more precise actions to be taken and resources to be provided to improve the operation of the system as well as student learning.

Educational indicator systems, using multiple indicators, fulfill the need to understand, monitor and enhance educational quality and for holding school accountable. Educational indicator systems are a mixture of outcome measures, organizational effectiveness measures, input measures, and student difference measures. Each of these areas has an impact on student learning as should be perceived as an integrated whole that works as a system.

Indicators can be established at the student level, the classroom level, the teacher level, and the school organizational level. Some indicator systems also include community context indicators. Each of these levels becomes a data set that can answer a number of questions about the operation of the school.

Characteristics of an effective indicator system include:

• Primary focus on the school level
• Use of multiple indicators of school performance
• Use of multiple measurement and data collection techniques
• Use of adaptive indicators that evolve over time
• Focus on school change and improvement over time.

For additional information on establishing educational indicator systems, please see the Educational Resources Information Clearinghouse for Assessment and Evaluation and search for educational indicators. This search will yield a number of practical examples and background articles. Also see the National Center for Education Statistics for the common core of data initiative that aims to systematize data terminology and collection methods across the states.

Building Tests To Support Instruction and Accountability

A Guide for Policymakers

Prepared By
The Commission on Instructionally Supportive Assessment

Convened By
American Association of School Administrators
National Association of Elementary School Principals
National Association of Secondary School Principals
National Education Association
National Middle School Association

October 2001

Members of the Commission on Instructionally Supportive Assessment

Eva L. Baker
Professor of Educational Psychology and Social Research Methods
Co-Director, National Center for Research
      on Evaluation, Standards, and Student Testing
Graduate School of Education and Information Studies
University of California, Los Angeles

David C. Berliner
Regents' Professor of Education
College of Education
Arizona State University

Carol Camp Yeakey
Professor of Urban Politics and Policy
Curry School of Education
University of Virginia

James W. Pellegrino
Distinguished Professor of Cognitive Psychology
Distinguished Professor of Education
University of Illinois at Chicago

W. James Popham (Chair)
Professor Emeritus
Graduate School of Education and Information Studies
University of California, Los Angeles

Rachel F. Quenemoen
Senior Fellow for Technical Assistance and Research
National Center on Educational Outcomes
University of Minnesota

Flora V. Rodríguez-Brown
Professor of Curriculum and Instruction/Reading, Writing and Literacy
University of Illinois at Chicago

Paul D. Sandifer (Ret.)
Consultant to the Office of Assessment
South Carolina Department of Education

Stephen G. Sireci
Associate Professor
School of Education
Center for Educational Assessment
University of Massachusetts, Amherst

Martha L. Thurlow
Senior Research Associate, Department of Educational Psychology
College of Education and Human Development
Director, National Center on Educational Outcomes
University of Minnesota

Commission members' affiliations do not denote institutional endorsement of the Commission's report.

EXECUTIVE SUMMARY

State-mandated accountability tests must be useful to educators concerned about improving the instruction of children.

The Commission on Instructionally Supportive Assessment, in support of this assertion, presents nine requirements for a new generation of statewide achievement tests. The Commission believes that tests written to these requirements will benefit students by providing educators with information they can use to improve the quality of instruction. At the same time, the tests will provide states with information to hold educators, schools, and school districts accountable for student performance.

Clearly, state policymakers who pass accountability legislation have in mind this dual outcome of assessing and improving student performance. But, all too often, while state-administered achievement tests measure performance, they have little value for instruction. This minimizes their usefulness in an accountability system that assumes information from tests will result in appropriate changes in instruction.

To address this problem, the Commission calls for states to use the nine requirements as steps to create responsible state assessment systems, including tests that improve both learning and accountability. Each requirement is supported by reasons for its importance.

The nine requirements are:

The Commission of nationally recognized experts in assessment, curriculum, and instruction was convened by five national associations representing administrators and teachers: American Association of School Administrators, National Association of Elementary School Principals, National Association of Secondary School Principals, National Education Association, and National Middle School Association.

Table of Contents

PREFACE

The individuals who prepared this report were asked to recommend ways that state-administered achievement tests could not only satisfy public demands for accountability, but also improve the instruction of children. I had the privilege of selecting the Commission's members from a list of nominees submitted by a coalition of administrator and teacher organizations. I used two criteria. First, I wanted different perspectives represented. Second, I wanted individuals who possessed recognized expertise in assessment and/or instruction. As a Commission, we functioned with complete autonomy. Members of the commission received no remuneration for their work, other than reimbursement for travel expenses. We appreciate the opportunity to consider independently this important educational issue.

W. James Popham
Commission Chair

FOREWORD

Educators who assume the tremendous responsibility of ensuring a quality education for all children can be effective only with the support and encouragement of the larger society. State-mandated standards are a form of encouragement. They signal to educators that the public supports educational improvement for every child, regardless of his or her background. Teachers and administrators agree that, when done well, state standards can increase academic rigor, strengthen curricula, and encourage student learning. But when the standards are linked to state tests, particularly those with high-stakes consequences, educators have understandable concerns.

All too often educators' legitimate concerns about state assessment are not heard by reform-minded policymakers. Worried that they'll be accused of running from accountability, teachers, principals, and superintendents publicly embrace state assessments while they fret privately about their negative impact on sound teaching and learning.

This dilemma brought together five associations that serve the needs of classroom teachers, school principals, and district superintendents. Our associations acknowledged that those who mandate state tests frequently believe our ties to our respective memberships negate the impartiality of our concerns. We then agreed that it would take an independent group of nationally recognized experts in assessment, curriculum, and instruction to attract the attention of state policymakers and urge them to reexamine their assessment systems. Only then, we concluded, would assessment systems be improved so that everyone-educators, policymakers, parents, and the public at large-could be confident that accountability requirements are linked to sound teaching and learning.

W. James Popham, noted research scholar and professor emeritus of the University of California, Los Angeles, was tapped to convene the Commission and serve as its chair. Commission members include nationally recognized authorities in assessment, curriculum, and instruction, who have brought a wide breadth of knowledge and experience to this effort.

The five associations enthusiastically embrace the result of the Commission's deliberations and applaud its bold assertion that a state's assessment system must meet nine requirements to be seen as credible. We recognize that the Commission makes no mention of what it might cost states to implement truly effective assessment systems. The costs will vary across states, of course. Some have already made major investments in their systems; others have a long way to go. But without an adequate assessment system, neither educators, parents, policymakers, nor the public will know if their expectations for educational quality are being met.

We thank Dr. Popham and the other Commissioners for their hard work and we appreciate their clearly written report. We urge our respective memberships to read the Commission's report. And then we ask them to use it together to begin anew the dialogue with state policymakers about the efficacy of their assessment systems.

Teachers, principals, and district superintendents will readily embrace accountability measures if they are tied to effective assessment systems designed and implemented to improve classroom instruction. Our hope is that policymakers will too.

Finally, we acknowledge our association colleagues who made this collaborative activity possible. Particular thanks go to Marcella Dianda of NEA for her leadership role. Others involved from inception through completion were Gail Gross of NAESP, John Nori of NASSP, and Cynthia Prince and Joe Schneider of AASA.

Paul D. Houston
Executive Director
American Association of School Administrators

Vincent L. Ferrandino
Executive Director
National Association of Elementary School Principals

Gerald N. Tirozzi
Executive Director
National Association of Secondary School Principals

Robert E. Chase
President
National Education Association

Sue Swaim
Executive Director
National Middle School Association

INTRODUCTION

This report evolved in response to the question: Why shouldn't state-mandated accountability tests also be useful to educators concerned about improving the instruction of children? Simply put, why can't state tests be designed so that the results are useful to educators as they revamp instruction to improve student achievement? Clearly, the policymakers who pass such legislation have this dual outcome in mind. But, all too often, federally mandated and state-administered tests seem to have little instructional utility, thus bringing into question their usefulness in an accountability system that assumes that information obtained from tests will result in appropriate changes in instruction.

The Commission on Instructionally Supportive Assessment was convened at the request of a coalition of administrator and teacher organizations. We accepted the coalition's request for assistance because we believe the current focus on educational testing offers an exceptional opportunity to create assessments that can help the nation's children learn better. If tests help teachers do a better job in the classroom, then they will truly be instructionally supportive. Moreover, we believe that such assessments can provide policymakers with the kind of meaningful evidence needed to satisfy today's educational accountability demands. Consequently, we have written this report specifically for state policymakers to help them establish educational policies that will lead to the development of tests supportive of both instruction and accountability.

We do not believe that the results of a single test should ever be used to make significant decisions that affect schools or students. The tests we recommend will, however, provide one important source of evidence for key educational decisions.

Our report contains nine requirements that must be met to ensure a responsible educational assessment system for the improvement of learning. Certainly, policymakers might satisfy these requirements in various ways. One approach would be to draw on the capabilities of state agencies, such as a state's department of education, to carry out certain of the required activities. Another would be to have the state issue a competitive request for proposals (RFP) to firms and individuals capable of carrying out one or more of the Commission-required activities.

The Commission has issued a separate report to assist state authorities who choose to rely on RFPs and external contractors to obtain necessary products and/or services. That report provides illustrative language that might be incorporated into RFPs to satisfy any of the Commission's nine requirements. (See Illustrative Language for an RFP To Build Tests To Support Instruction and Accountability, 2001, available online from each convening association.)

The Commission recognizes that most states have intended to create assessment systems aimed at improving instructional quality. This document is an effort to help ensure that such a goal is reached.

Requirements for the Development of Instructionally Supportive Assessments

The Commission believes that states will create responsible assessments for the improvement of students' learning if they implement the nine requirements we describe on the following pages.

Requirement 1
A state's content standards must be prioritized to support effective instruction and assessment.

Because --

• educators in many states cannot adequately address within the amount of time available for instruction the large number of content standards that are supposedly measured by state tests;
• state tests often do not adequately assess all of the content standards, and frequently center on standards that are easiest to assess; and
• state tests rarely provide educators with the kind of information they need to improve instruction.

Nearly every state has developed two kinds of standards. Content standards focus on knowledge and skills educators are expected to teach. Performance standards focus on students' proficiency in demonstrating the knowledge and skills de