NWEA Spark Community

Obtaining a valid test score from a student requires more than just a good test.  It also requires an engaged student who devotes enough effort to the test to demonstrate his true level of proficiency. Without sufficient effort, regardless how good the test is, a student’s test score is likely to markedly underestimate what he knows and can do.  In this case, the student’s score is said to reflect low individual score validity (ISV) due to low test-taking effort.  This means that the score should not be considered a trustworthy indicator of the student’s proficiency level.  It also means that any growth score that is calculated using a low ISV score will be distorted and should be interpreted with caution.

No assessment is immune to this problem.  And despite every effort made to ensure the highest quality of assessments, we can’t control the behavior of the students who are testing.  Instead, we try to identify instances of students not giving their best efforts.  This allows us to identify scores with low ISV.

What Is Known About Non-Effortful Test Taking in MAP Assessments

Researchers at NWEA have pioneered methods for identifying test events that reflect low ISV.  Using these methods, we have established and refined criteria for classifying scores as reflecting low ISV.  Based on these criteria, we have discovered a number of important relationships about student engagement.  These include findings that:

• Boys exhibit scores with low ISV about twice as often as do girls.  For some reason, this finding surprises no one.
• Low ISV is about twice as likely to occur on a reading test event than on a math test event.  This is because reading test items tend to contain more words, which students view as requiring more effort to answer.
• Scores with low ISV are more common during fall testing than during spring testing.  This is a bit counterintuitive, as some partners believe that spring MAP testing (coinciding with the state accountability testing and other end-of-school-year demands) is more vulnerable to low student effort.
• The prevalence of low ISV scores increases with grade.  In one study, we found that about 1% of the MAP scores show low ISV at grade 2, with a gradual increase to about 15% in grade 9.
• The time of day that MAP is given is important.  The percentage of low ISV scores gradually increase during the day, with students tested late in the day exhibiting low ISV about twice as often as those tested at the beginning of the day.

What Can Educators Do About the Problem?

Educators should be mindful that how, when, and where a student is administered MAP matters.  Students should be given encouragement from their teachers to give their best effort, and the importance of getting valid MAP scores should be emphasized.  MAP should be administered in a quiet setting that is free of distractions, and there should not be incentives to students to rush through the test (such as going to lunch or recess immediately afterward).  Ideally, students should be testing at the beginning of the day (though we realize that this will not always be possible).  The general goal is to administer MAP in such a way that maximizes student engagement throughout their test events.

What Can NWEA Do About the Problem?

We at NWEA are committed to providing the best possible information to educators about their students’ academic growth.  Part of that commitment involves our being able to identify low ISV scores.  Such scores should be identified on our score reports so that our partners are aware of them.  In addition, we are also working on developing methods of adjusting RIT scores for the amount of non-effortful behavior that occurs during test events.  Finally, we are trying to develop a “smarter” type of MAP that can preempt disengaged test taking by identifying it as it happens and notifying proctors or by displaying encouraging messages to the student.

Steve Wise is a Senior Research Fellow at NWEA.