For over four decades, the National Assessment of Educational Progress (NAEP) has provided information on the achievement and performance of students in the United States. NAEP, known as “The Nation’s Report Card,” is the largest continuing, nationally representative measure of what U.S. students know and can do in various subject areas. NAEP is a congressionally mandated program, administered by the National Center for Education Statistics at the U.S. Department of Education. The National Assessment Governing Board sets policy for NAEP, and its responsibilities include the assessment schedule, frameworks, and achievement levels for NAEP.
Achievement levels have become a powerful way to communicate student achievement on an assessment like NAEP writing at grade 4 because achievement levels interpret test performance with reference to cut scores that quantitatively define ordered categories of achievement such as basic or proficient. An important source of evidence used by policymakers to establish achievement levels is the cut score recommendations. Cut scores are the outcome of a facilitated process, called an achievement levels-setting panel or standard setting meeting, eliciting judgments from experts related to the test content and the knowledge, skills, and abilities of the test takers.
The National Assessment Governing Board awarded a contract to Pearson Education to develop achievement levels on the NAEP writing assessment in grade 4. Pearson Education is conducting achievement levels-setting field trial, pilot, and operational studies to develop the achievement levels for reporting performance of the nationally representative sample of 4th-grade students who are participating in the 2017 NAEP grade 4 writing assessment.
A pilot meeting will be held in November of 2017 to determine whether modifications for training, instructions, materials, timing, and logistics will be needed for the operational meeting. The pilot meeting will provide an opportunity for facilitators to practice the process before moving to the operational meeting and also provide feedback that will be used to validate or modify procedures utilized in the operational study. An operational meeting will be held in February 2018 at the same venue as the pilot meeting for the purpose of determining final cut score recommendations to the Governing Board.
A total of 22 panelists will be recruited to participate in the achievement levels-setting pilot study meeting and a total of 33 panelists will be recruited to participate in the achievement levels-setting operational meeting. The pilot panelist pool will include: 12 classroom teachers currently engaged in grade 4 writing instruction, 7 members of the general public, and 3 non-teacher educators. The operational panelist pool will include: 18 classroom teachers currently engaged in grade 4 writing instruction, 10 members of the general public, and 5 non-teacher educators. The objective of the recruitment plan is to produce panels with broadly representative, well-qualified panelists to participate in the pilot and operational activities. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation. Panel members need not have prior experience with this kind of study. They will be trained in the achievement levels-setting methodology by expert facilitators.
Selected panelists will participate in group sessions to implement the procedures for setting achievement levels. The pilot meeting is scheduled for November 6-9, 2017 in Atlanta, Georgia. The operational meeting is scheduled for February 12-15, 2018 in Atlanta, Georgia in the same venue as the pilot meeting.
Panelist nominators will be obtained through the allied organizations that were involved in the steering of the NAEP Writing assessment framework, provided feedback on the framework, or have a strong background in providing professional development in writing education. These allied organizations will be supplemented by additional organizations to increase representation and to increase the potential pool of candidates for the panels.
Each nominator will be asked to nominate up to 4 candidates for the NAEP grade 4 writing pilot and operational achievement level-setting panel based on the qualifications presented below.
Panelists nominated for the classroom teacher category must meet the following qualifications: (a) At least five years of overall teaching experience; AND (b) At least two years of experience teaching writing at grade 4; AND (c) Judged to be “outstanding” in their professional performance by a nominator. Teacher nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.
Panelists nominated for the non-teacher educator category must meet the following qualifications: (a) Non-teacher educational staff at secondary schools with education and/or experience with writing at grade 4; OR (b) Curriculum director or content specialist at secondary school or state department of education with education and/or experience in writing at grade 4, OR; (c) Postsecondary teacher education faculty teaching courses in elementary writing instruction. Non-teacher educator nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.
Panelists nominated for the general public category must meet the following qualifications: (a) An expert in writing; AND (b) Not a former educator; AND (c) Familiar with students in grade four (e.g., as a parent or volunteer). General public nominees will be asked to provide information regarding their qualifications and experience for serving on the panel. Panels will reflect an overall balance of gender, race/ethnicity, and type of institutional affiliation.
The selected panelists will be asked to read brief materials provided online prior to their participation in the pilot and operational achievement levels-setting panels.
Participation in the NAEP grade 4 writing pilot and operational panels represent an exceptional contribution to writing education in the United States. Expenses for travel, meals, and lodging will be paid for all panelists in compliance with federal travel regulations and substitute pay will be provided for participating teachers. In addition, panelists will receive an honorarium of $500.00 for participation in either the pilot or operational panel meeting.
For further information contact:
Senior Project Manager