FAQ: Improving test invalidation due to rapid-guessing

Document created by Community User on Oct 16, 2018Last modified by Community User on Nov 14, 2018
Version 6Show Document
  • View in full screen mode
Introduction
Frequently asked questions about how NWEA plans to address test invalidations due to rapid-guessing concerns.

Information

What happened during the recent fall test season related to rapid-guessing and test invalidation?

MAP Growth partners saw a higher number of invalidated tests because of excessive rapid-guessing after the July 2018 MAP Growth product update. This update introduced a new invalidation rule that automatically invalidated the test if a student rapid-guessed on 30 percent or more of the total number of questions received. This caused confusion and frustration to many partners during fall testing. We know this was not rolled out effectively to our partners. We did not provide adequate supporting documents, guidance, and necessary time for partners to prepare for retesting and train proctors accordingly.

While we continue to alert proctors to rapid-guessing, starting October 31, 2018, MAP Growth will not invalidate tests due to excessive rapid-guessing. We are updating help documents, training support staff, and providing early communication to partners starting with an email on October 17 from our CEO, Chris Minnich, with the goal of re-introducing invalidation at a future date.

Your partnership and feedback are essential in the process as we work to improve our assessments and make the information they provide timelier and more useful to you, your teachers, and your students.

 
What changes are being made for the upcoming winter test season?

Starting October 31, 2018, MAP Growth will not invalidate tests due to excessive rapid-guessing. All other test invalidation rules remain unchanged.

The following reports will not display invalidated tests due to excessive rapid-guessing:
These changes will not, however, affect the proctor alerts due to rapid-guessing during testing and will not change previous MAP Growth scores.

 
What if districts want to see which students rapid-guessed on 30% or more of the total number of questions received?

Starting November 12, 2018, the Grade Breakdown Report will display the percent of questions rapid-guessed for each student. The Grade Breakdown Report can only be accessed by the Administrator, School Assessment Coordinator, or District Assessment Coordinator roles in MAP Growth. This information will be shown under the header of % of Disengaged Responses in column K.
Grade Report test invalidation

 
Will the Student Profile Report display disengagement data?

Yes, the Student Profile Report will continue to display Percentage of Disengaged Responses and Estimated Impact of Disengagement on RIT.
Screenshot of a portion of the Student Profile report showing various testing statistics including RIT score, % of disengaged responses, and estimated impact of disengagement on RIT score

 
Will proctors still receive an alert when students rapid-guess during the live testing session?

Yes. Proctors will receive an alert when students rapid-guess. Note, many proctors are not at their computers at all times watching this screen during testing. The proctor screen refreshes every 60 seconds, indicated in the image below, but students can rapid-guess before the proctor is alerted because of the 60 second refresh window. In short, the alert is not real time, but delayed.
Screenshot of a notification on the proctor screen showing that a number of students are disengaged

Proctors can click the orange Student Disengagement box to expand the information.
Screenshot showing students with disengagement alerts. You can click the alert for each student to dismiss the alert. You can also click the notification that shows how many students are disengaged to dismiss all disengagement alerts.

 
What improvements are being made after winter testing?

Full plans for re-introduction are not finalized but will include more robust proctor tools such as the ability for a proctor to auto-pause a student test.

Additionally, we will update our in-product help to be more thorough and provide professional learning and onsite training on the topic of student disengagement so that you have full support for a successful testing session across your schools.

Look for an update from NWEA on plans in January.

 
Who can I contact for advanced inquiries?

Please contact NWEA Partner Support.

 
Why is NWEA turning it off, then back on again in the future? Is NWEA considering these tests valid now? Is there any impact on data validity?

We believe that the invalidation rule is important in that it provides a more valid and reliable test experience and ensures you have the highest quality data possible to make decisions that impact students and their learning.

 
Is NWEA going to change how often the proctor screen refreshes?

We are reviewing the impact of adjusting the 60 second refresh but do not plan to adjust it by the winter test season.

 
Will students who are rapid-guessing receive a warning in MAP Growth?

We are reviewing this request but do not plan to add this request by the winter test season.

 
Is it meaningful to compare fall to winter scores due to the changes coming out October 31, 2018?

MAP Growth data comparison from fall to winter continues to provide you valuable information about student growth to inform instruction at the district, classroom and student levels.

While, there is no change to those students that did not rapid-guess, we recommend you use caution when evaluating students who did rapid-guess. As you compare individual student scores to inform instruction, it is always best to use the valid score as the basis for instruction. Using a score that was generated through rapid-guessing may provide lesser quality information.

Starting November 12, 2018, the Grade Breakdown Report will display the percent of questions rapid-guessed for each student. This report will allow you to identify students who reached or exceeded the 30 percent threshold.

See also:

Article Number
3688

Attachments

    Outcomes