Product Update

MAP Growth K-2 Scale Maintenance FAQ

Blog Post created by Product Update on Apr 15, 2020

Introduction

As a part of the 2020 Norms release, NWEA will perform scale maintenance on the MAP Growth K-2 test. More specifically, NWEA is updating item calibrations on the MAP Growth K-2 test, which will result in adjustments to K-2 scores starting in July 2020. We have also revised our MAP Growth grade-level test transition guidance (K-2 to 2-5) to provide clearer guidelines for our partners to use across this test transition period. Taken together, the scale maintenance effort and updated guidance will result in a significant reduction in score drops when students transition between the MAP Growth K-2 and 2-5 tests. This FAQ provides answers to common questions NWEA partners may have about the scale maintenance efforts, the subsequent effect on student scores on the MAP Growth K-2 test, and some of the implications for partners.

FAQs

General:

  • What is scale maintenance and what does it mean?
    • “Scale maintenance” refers to work being done on the MAP Growth K-2 tests. This involves evaluating and updating item calibrations on the MAP Growth K-2 test (meaning re-evaluating the RIT difficulty of all items in the MAP Growth K-2 item pool), and in turn, re-estimating student scores on the K-2 test. As a result of this work, and if partners follow the NWEA MAP Growth grade-level test guidance, this should result in significant reductions of declines in student RIT scores when transitioning from the K-2 to 2-5 test.

 

  • What specific test/scale are you talking about?
    • NWEA will perform targeted scale maintenance on the MAP Growth K-2 test in July 2020. No changes will be made to the 2-5 or 6+ tests.

 

  • When will the scale maintenance take place?
    • The MAP Growth K-2 scale maintenance will occur in conjunction with the July 2020 MAP Growth software release, currently scheduled for July 24, 2020.

 

  • Are there any timing considerations for partners who may be testing during this time?
    • NWEA strongly encourages partners to avoid, when possible, having a testing window run across the July 2020 release (i.e. starting testing before July 24, 2020 and completing testing after that date). This is because the 2020 norms update and scale maintenance initiatives that are part of the July MAP Growth release will be implemented in the middle of the summer testing window. This situation could cause significant reporting issues due to the fact that tests taken before and after the MAP Growth July 2020 release would reflect pre- and post-maintenance adjustments.

 

  • Where can I learn more details about the research that went into this scale maintenance process?
    • A summary providing an overview of the scale maintenance initiative, including a summary of the rationale, methodology, results, and implications of this work for partners may be found here. We recommend this as your primary source of information on this topic.
    • If you still have questions that remain unanswered, please contact your Account Manager.

 

  • What exactly does the scale maintenance work entail?
    • The primary outcome of this scale maintenance work is the adjustment of item calibrations for certain items across the MAP Growth K-2 item pool, which in turn will result in adjustments in scores on the K-2 test from July 2020 forward. This will result in improved transitions for students moving from the MAP Growth K-2 to 2-5 test and will minimize the magnitude of RIT score drops often observed by partners when their students make this transition. For more information on this work, please see the 2020 MAP Growth K-2 Scale Maintenance Research Summary.

 

  • What does it mean to adjust the calibration of MAP Growth K-2 items?
    • An item’s calibration informs its difficulty level; that is, this process identifies the point on the RIT scale where approximately 50% of students answer the item correctly, and 50% answer the item incorrectly. This information, in turn, informs to which students should that item be administered. The goal of an adaptive test is to match items to students at a difficulty level commensurate with students’ achievement level. In other words, an item that calibrates at a 200 level would be appropriate to administer to a student with an estimated RIT score of 200. The aim of this scale maintenance work is to ensure that calibrations for items in the MAP Growth K-2 item pool accurately reflect their difficulty level.

 

  • What is the impact of the calibration adjustments specific to this K-2 scale maintenance process?
    • Adjustments in item calibrations on the MAP Growth K-2 test vary by grade, subject, and achievement level. In general, the more difficult the item (i.e. higher RIT score), the greater the downward adjustment of the difficulty of the item. In some grade/subject areas, such as kindergarten mathematics, the adjustment was the opposite direction, with less difficult items calibrating slightly higher (i.e. more difficult).

 

  • Does this mean my student’s previous MAP Growth K-2 test scores were wrong?
    • No – previous MAP Growth K-2 scores were reflective of the item calibration and scoring approach in use at the time at which those scores were produced, and those approaches were consistent with best practices at that time. Subsequent improvements in approach should lead to improved accuracy of student scores when these changes are implemented in July 2020. Prior MAP Growth K-2 scores were not “wrong”— the changes that have been made will lead to improvements in the estimation of student achievement, and in turn, reduce or mitigate altogether RIT score drops often observed when students transition from the K-2 to 2-5 test.

 

  • Did some students taking the MAP Growth K-2 in the past have artificially high scores?
    • Score inflation can occur when there are not enough items available in the item pool to sufficiently challenge a student. In those instances, students will get easier items than they are capable of answering, which can lead to increases in the percent of items answered correctly (which typically is 50% on the MAP Growth assessments). This has been observed for second graders who take the MAP Growth K-2 test, especially when the students are high achieving and likely able to read independently and engage with second-grade level content. These are students who would be better served by taking the MAP Growth 2-5 test and would have a significantly higher number of items available to them commensurate with their achievement level. For more information on when to transition a student to the MAP Growth 2-5 test, please reference the MAP Growth grade-level test guidance and the Expanded guidance on test administration for second-grade students.

 

  • What kind of impact can we expect to see? Will student scores on the MAP Growth K-2 assessments just be lower? If so, how much will they drop?
    • As a part of this scale maintenance process, the calibrations of MAP Growth K-2 items will be reviewed and adjusted as needed, and this will result in updates in scoring for MAP Growth K-2 results starting in July 2020 forward. The magnitude of student score changes are shown in the table below, which indicate that on average, student scores on the K-2 test will be lower as students get older and higher achieving. For example, second graders taking the K-2 mathematics test in the spring would see an average change in score of just over 6 points (lower) compared to student results from the pre-maintenance period.
    • Important note – prior (historical) scores will not be changed in the NWEA reporting system. So, while there may be some differences observed for students who test(ed) on the K-2 test prior to and after this scale maintenance process, students will not see any changes to scores attained from July 2020 forward.

 

Descriptive statistics of student RIT scores after re-scoring with adjusted item RIT values

 

  • Are you actually changing the items?
    • Items are not being changed as a part of this process. Item calibrations (i.e. item difficulty) are simply being updated for those items where updates are needed.

 

  • Are both the math and reading scales undergoing this specific K-2 scale maintenance process?
    • Both math and reading within the MAP Growth K-2 test are included in this initiative.

 

  • Are math and reading equally impacted … or are there differences?
    • Please see the table above that shows the magnitude of changes across mathematics and reading (and grade and term)

 

  • Will there be changes to the MAP Growth K-2 learning continuum based on the item recalibration?
    • Yes, some RIT bands will shift. Any changes will be reflected in the learning continuum and reporting after the July 2020 software release.

 

  • How will/should this impact instructional decisions?
    • This maintenance work won’t have any impact on instructional decisions made for students on the basis of their scores. Teachers can still use their MAP Growth K-2 data in the same manner as they always have.

 

  • Will the 2020 norms take the scale maintenance changes into account?
    • The 2020 norms, also being implemented in July 2020, will reflect the changes made as a result of the scale maintenance process.

 

  • If the difficulty score (i.e. RIT score) for the more difficult items on the MAP Growth K-2 test is going down, does this mean that there will be fewer high performing students?
    • No, this should not influence the proportion of students identified as high performing. The greatest changes in RIT scores are for students in second grade who are high achieving who are likely able to read independently. This work reinforces the need to make sure that students are transitioned to the correct test at the right time, so that decisions about which students are “high-performing” are based on results from a test that most closely reflects the instructional and developmental needs of individual students.

 

  • Will there be any scale maintenance work on the MAP Growth 2-5 or 6+ tests?
    • The targeted July 2020 scale maintenance work will not impact item calibrations or student scores on the 2-5 or 6+ tests. Routine scale maintenance is a continuous process across all assessments.

 

Reporting and Historical Data:

  • When will adjusted MAP Growth K-2 scores be implemented and start showing in reports?
    • The MAP Growth system will begin to display adjusted MAP Growth K-2 scores following the July 2020 software release.

 

  • Will the historical MAP Growth K-2 reporting/data be rescored to reflect these changes?
    • No, historical MAP Growth K-2 scores will not be updated in the NWEA reporting system.

 

  • Why didn’t you retroactively implement this scale maintenance in all historical reports?
    • The feedback we received from a number of partners was to avoid rescoring historical MAP Growth K-2 data in the NWEA reporting system. 

 

  • Will partners have any way to look at rescored MAP Growth K-2 historical data?
    • Yes, a new conversion file (“K-2 Scale Maintenance RIT Score conversion file.csv”) will be made available in the MAP Growth reporting system as of the August 2020 MAP Growth software release. Only users with school and administrator level access will be able to access this new conversion file. This conversion file will enable partners to see what the scores from K-2 tests taken before July 2020 would look like if they were adjusted in a manner consistent with K-2 scores from July 2020 forward. This file will also include 2020 normative achievement data (i.e. percentiles) associated with the rescored historical data.

 

  • How will reports be impacted where previous MAP Growth K-2 results and growth will show?
    • Reports from July 2020 forward will not be affected by these changes.
    • NWEA recommends avoiding the creation of reports that include MAP Growth K-2 test events from before and after this scale maintenance work was implemented (such as an ASG report from Fall ’19 to Fall ’20, which would span the July 2020 scale maintenance implementation).

 

Partner Guidance and Considerations - Data Usage and Comparisons for 2020/21 school year and beyond

  • Can I continue to compare RIT scores or growth data before and after this scale maintenance?
    • Scores on the MAP Growth K-2 test should not be compared across the July 2020 implementation of this scale maintenance work. To illustrate, student K-2 scores attained prior to July 2020 will rely on different item calibrations and scoring methodology than K-2 scores attained after July 2020. Because of this, comparisons should not be made between K-2 scores attained before and after July 2020.
    • Because historical scores (tests taken prior to July 2020) will not be adjusted in the MAP Growth reporting system, we strongly encourage partners to abstain from measuring growth for students in the first and second grade on the MAP Growth K-2 test across this scale maintenance period (July 2020), especially if student growth is used for high-stakes decision for students, teachers, or schools.
    • From July 2020 forward, there should be no issue examining longitudinal MAP Growth K-2 data as long as all data being compared are from July 2020 and after.
    • This work has no effect on longitudinal data on the MAP Growth 2-5 and 6+ tests.

 

  • What term pairs can I use for measuring growth for MAP Growth K-2 test takers?
    • For the 2020/2021 school year, we recommend that partners only measure student growth from fall 2020 forward. After the initial year of the scale maintenance implementation, partners can measure growth across any term pair, so long as growth is not measured across the July 2020 transition period (i.e. growth does not include MAP Growth K-2 data from prior to July 2020).

 

  • What if spring-to-spring comparison is part of our metrics or tracking? How would NWEA recommend we measure when comparing spring 2020 to spring 2021?
    • In the initial year, we instead recommend that growth on the MAP Growth K-2 test be measured from fall 2020 forward. Starting in spring 2021, there should be no issue in measuring spring-to-spring growth moving forward.

Outcomes