NAEP Grade 4 Writing Achievement Levels Setting Field Trial

NAEP Grade 4 Writing Achievement Levels Setting Pilot & Operational Studies

Frequently Asked Questions

What is NAEP?

For over four decades, the National Assessment of Educational Progress (NAEP) has provided information on the achievement and performance of students in the United States. NAEP, known as “The Nation’s Report Card,” is the largest continuing, nationally representative measure of what U.S. students know and can do in various subject areas. NAEP is a congressionally mandated program, administered by the National Center for Education Statistics at the U.S. Department of Education. The National Assessment Governing Board sets policy for NAEP, and its responsibilities include the assessment schedule, frameworks, and achievement levels for NAEP.

What is an achievement levels-setting panel?

Achievement levels have become a powerful way to communicate student achievement on an assessment like NAEP writing at grade 4 because achievement levels interpret test performance with reference to cut scores that quantitatively define ordered categories of achievement such as basic or proficient. An important source of evidence used by policymakers to establish achievement levels is the cut score recommendations. Cut scores are the outcome of a facilitated process, called an achievement levels-setting panel or standard setting meeting, eliciting judgments from experts related to the test content and the knowledge, skills, and abilities of the test takers.   

Who is leading the achievement levels-setting panels for NAEP grade 4 writing?

The National Assessment Governing Board awarded a contract to Pearson Education to develop achievement levels on the NAEP writing assessment in grade 4. Pearson Education is conducting achievement levels-setting field trial, pilot, and operational studies to develop the achievement levels for reporting performance of the nationally representative sample of 4th-grade students who are participating in the 2017 NAEP grade 4 writing assessment.

How do the pilot and operational meetings relate to each other?

A pilot meeting will be held in November of 2017 to determine whether modifications for training, instructions, materials, timing, and logistics will be needed for the operational meeting. The pilot meeting will provide an opportunity for facilitators to practice the process before moving to the operational meeting and also provide feedback that will be used to validate or modify procedures utilized in the operational study. An operational meeting will be held in February 2018 at the same venue as the pilot meeting for the purpose of determining final cut score recommendations to the Governing Board.

Who will be participating in the panels?

A total of 22 panelists will be recruited to participate in the achievement levels-setting pilot study meeting and a total of 33 panelists will be recruited to participate in the achievement levels-setting operational meeting. The pilot panelist pool will include: 12 classroom teachers currently engaged in grade 4 writing instruction, 7 members of the general public, and 3 non-teacher educators. The operational panelist pool will include: 18 classroom teachers currently engaged in grade 4 writing instruction, 10 members of the general public, and 5 non-teacher educators. The objective of the recruitment plan is to produce panels with broadly representative, well-qualified panelists to participate in the pilot and operational activities. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation. Panel members need not have prior experience with this kind of study. They will be trained in the achievement levels-setting methodology by expert facilitators.

Where/when will the achievement levels-setting panels take place?

Selected panelists will participate in group sessions to implement the procedures for setting achievement levels. The pilot meeting is scheduled for November 6-9, 2017 in Atlanta, Georgia. The operational meeting is scheduled for February 12-15, 2018 in Atlanta, Georgia in the same venue as the pilot meeting.

How will the nominators for panelists be selected?

Panelist nominators will be obtained through the allied organizations that were involved in the steering of the NAEP Writing assessment framework, provided feedback on the framework, or have a strong background in providing professional development in writing education. These allied organizations will be supplemented by additional organizations to increase representation and to increase the potential pool of candidates for the panels.

What will be the expectations from the nominators?

Each nominator will be asked to nominate up to 4 candidates for the NAEP grade 4 writing pilot and operational achievement level-setting panel based on the qualifications presented below.

How will the classroom teacher panelists be selected?

Panelists nominated for the classroom teacher category must meet the following qualifications: (a) At least five years of overall teaching experience; AND (b) At least two years of experience teaching writing at grade 4; AND (c) Judged to be “outstanding” in their professional performance by a nominator. Teacher nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.

How will the non-teacher educator panelists be selected?

Panelists nominated for the non-teacher educator category must meet the following qualifications: (a) Non-teacher educational staff at secondary schools with education and/or experience with writing at grade 4; OR (b) Curriculum director or content specialist at secondary school or state department of education with education and/or experience in writing at grade 4, OR; (c) Postsecondary teacher education faculty teaching courses in elementary writing instruction. Non-teacher educator nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.

How will the members of general public panelists be selected?

Panelists nominated for the general public category must meet the following qualifications: (a) An expert in writing; AND (b) Not a former educator; AND (c) Familiar with students in grade four (e.g., as a parent or volunteer). General public nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.

What preparation will be needed prior to participation in the panels?

The selected panelists will be asked to read brief materials provided online prior to their participation in the pilot and operational achievement levels-setting panels.

How will the panelists be rewarded for participation in the NAEP grade 4 writing pilot and operational meetings?

Participation in the NAEP grade 4 writing pilot and operational panels represent an exceptional contribution to writing education in the United States. Expenses for travel, meals, and lodging will be paid for all panelists in compliance with federal travel regulations and substitute pay will be provided for participating teachers. In addition, panelists will receive an honorarium of $500.00 for participation in either the pilot or operational panel meeting.

Where can I find more materials on NAEP Writing?

The 2017 NAEP Writing Framework was used for developing the assessment. The Writing Framework can be found here. Results from the 2011 NAEP Writing assessment at grades 8 and 12 can be found here.

For further information contact:
Julie Downey
Senior Project Manager
Pearson
319-341-5379

TOP