FAQ: Rapid-guessing, disengagement, and invalidated tests

Document created by Community User on Sep 10, 2018Last modified by Community User on Sep 25, 2018
Version 6Show Document
  • View in full screen mode
Introduction
Frequently asked questions about rapid-guessing and student test engagement, including tests that are invalidated due to a high level of rapid-guessing.

Information

How does NWEA determine which MAP Growth assessments to invalidate, in general?

NWEA has used a set of rules to ensure that test events are accurate reflections of student achievement since we began computerized adaptive testing nearly two decades ago. These rules are based on valid ranges for several measures, including test time, standard error of measurement and RIT score.

To further improve the accuracy of our assessments, and in response to partner feedback, we added rapid guessing detection in 2017. At this time, we also included an alert to proctors during a live testing session to enable them to intervene.

 
What are the MAP Growth test invalidation reasons?

All invalidation reasons occur at the submission and completion of the MAP Growth test and not before. 

MAP Growth test invalidation reasons:For details about invalid tests, see:

 
What is rapid-guessing behavior?

When a student answers an item very quickly—usually much faster than it would take to read and understand the question—that type of response is called rapid guessing. Such responses are important to identify because they do not provide credible information about what the student knows and can do. Worse, they tend to distort a student’s score downward.

 
When is a test invalidated due to rapid guessing?

Test events in which at least 30% of the item responses are rapid guesses are marked invalid. The invalidation reason appears on the end of test screen if the district has enabled the end of test screen. Tests that have been invalidated for any reason (including rapid guessing) appear on the Class and Grade reports in grey. See Grayed out (low lighted) test results.

See also: Why does NWEA invalidate MAP Growth tests for rapid guessing?

 
Why am I seeing a higher number of student test invalidations?

Since the July 2018 update in MAP Growth, tests events in which at least 30% of the item responses are rapid guesses are marked invalid. Before the July update, students may have been rapidly guessing at high levels, but this data was not reported, and tests were marked valid.

 
Where are rapid guessing metrics reported and clearly marked?

For students with valid tests, their rapid guessing metrics (if any) are reported in the Student Profile report. The percentage of disengagement and the impact to RIT appear in both the subject tab and the Growth Over Time module at the bottom of the report. 

When rapid guessing results in an invalidated test, the test appears on both the Class and Grade reports, along with a reason code of "high level of rapid guessing.”

Reports that do not clearly indicate invalid tests
Two operational reports show invalid tests but do not provide the reason code. Do not use them in determining which students had a “high level of rapid guessing”. The Students without Valid Test Results report lists students with incomplete, suspended tests, or completed tests that are invalid, but does not show the reason code for invalid tests. Similarly, the Test Event by Status report shows the number of invalid tests, but does not list the reason code.

 
How often does rapid-guessing occur that triggers the 30% threshold?

Test events that trigger the 30% rapid-guessing rule are not very common. Our studies indicate that scores would be invalidated for disengagement only about 2% of the time in math, and between 2-5% of the time in reading.

See also: Why does NWEA invalidate MAP Growth tests for rapid guessing?

 
Is rapid-guessing working as expected in MAP Growth? 

We have evaluated data from multiple partner organizations, and to date have not found any example of this not working properly.

 
Would technical issues impact rapid-guessing behavior?

No. Technical issues do not impact the functionality of our disengagement detection features, nor do they impact how disengagement and rapid-guessing are recorded and reported in MAP Growth.

 
Can we opt out or turn off the rapid guessing function?

No. We have included this function because we know it increases the accuracy of student data in MAP Growth assessments. Having accurate student data is critical in supporting growth and instruction.

See also: Why does NWEA invalidate MAP Growth tests for rapid guessing?

 
I’ve heard of Student Engagement, how is rapid-guessing behavior related?

Student Engagement is the common term used among educators and academics pertaining to the degree to which a student is generally engaged with school. This refers to such behaviors as paying attention, taking notes, listening, asking or answering questions, doing homework, or reacting (laughing, crying, shouting, etc.). Rapid-guessing behavior refers to the student’s disengagement in the specific context of taking an assessment.

Preliminary research shows that a student who is disengaged from school also tends to show disengagement when taking a test. However, some students who are typically engaged in school might not be engaged during a test; for example, if they are sick, fatigued, or bothered by some personal problem.

 
What is the impact of disengagement on the student’s score?

Rapid guessing tends to deflate a student’s score. We provide a metric on the Student Profile report called Estimated Impact of Student Disengagement on RIT which shows how different the score potentially would have been if the student had been fully engaged during the test.

For a detailed explanation, see the following page in the MAP Help Center: How Disengagement Affects Scoring.

 
When students show less than 100% engagement, how do we interpret their scores?

Refer to our guidance document found on NWEA.org: NWEA Guidance on the Interpretation and Use of New Engagement Metrics in MAP Growth Reports.

 
How does rapid-guessing behavior impact longitudinal data?

Rapid guessing impacts growth scores, either in a positive or negative direction (depending on when the student became disengaged). Rapid guessing also helps to explain instances of negative growth. For more information, see the following blog post written by the leading researcher in the field of student disengagement, Steve Wise: How Does Test Engagement Affect RIT Score Validity?

Additional blogs from Research:

 
If proctors are able to intervene, what do you recommend?

If proctors are able to intervene after receiving an alert in the proctor console, they should pause the student’s test and speak with them before resuming their test. For more information, see How to Intervene in the MAP Help Center.

 
What tips are available to ensure students are more engaged during MAP Growth testing?

See Tips to Prepare Students for MAP Growth in NWEA Connection. You must log in to access the PDF. See How do I create an account and sign in to NWEA Connection?

 
Where MAP Growth is used in a high-stakes testing environment with limited to no retesting allowed, what do I say when the test is invalidated?

Districts typically establish policies that limit or restrict re-testing to protect students from over-testing and to discourage the practice of “fishing” for high scores. However, in the rare case of invalidated test events due to rapid guessing, there is no test score available for the student. Consult with the appropriate school administrator to determine how to proceed when this happens and ask if non-scored students could re-attempt testing, if appropriate.

Be proactive and try to prevent disengagement during testing. Monitor the proctor console and intervene when students trigger the rapid-guessing alert.

Article Number
3671

Attachments

    Outcomes