Next to Artificial Intelligence (AI) scoring of student responses, I think the advancement that has changed performance assessment more than any one thing is distributed scoring. Recently, we have seen changes made to education and assessment as we adapt to a new way of living. As we look to the future, continual change is certain. I wanted to share my experience going through one of the more significant changes to performance scoring during my career and reflect on the benefits, which are easy to see now.
When I think back to a time of hundreds of scorers in a room, we had scoring directors training and monitoring a large group and team leaders supporting smaller groups of scorers for days or even weeks. I often wonder “How in the world did we do it?” As we move forward into this new “normal” the old model seems impossible.
At a previous organization, in which I was leading, we transitioned from in person, paper-based scoring to online distributed scoring. We made that choice based on projected, and now realized, benefits. Moving forward, these decisions will be made more out of necessity.
Below, I highlight a few of my initial doubts and also the realized benefits of this transition. Someday, when we are able to reflect back upon this period and all of the drastic changes that have been made to education, we will have more success stories to share.
Improving Efficiency and Integrity with Real-Time Validity Measurement
One of the most valuable benefits of online scoring is being able to administer validity. Truly- blind and interspersed at whatever interval one chooses, the scorers are not required to score twenty validity responses at one particular time but can be monitored throughout the day with the validity results available immediately.
When first moving to a distributed model, I was concerned it would move the scorer another level away from the student. However, I came to see the benefits for both students and teachers were significant. With distributed scoring, one of the main goals has been taking a very subjective process and making it as objective as possible. Distributed scoring not only makes sense logistically by improving efficiencies, it also improves the integrity of the scoring. There was no longer a need for runners-distributing packets to scorers, picking up completed packets and filing them at a warehouse until scoring was complete. By the end of scoring, the packets were covered with coffee stains, they had frayed edges and everyone dreaded them.
Creating Consistent, Un-Biased Training
While working to support SAT scoring at The College Board, essays were required to be scored by teachers. There was no question. Papers could no longer be scored at a physical site. We changed our process and instead of hundreds of scorers in one room, we had hundreds of scorers scattered across the country. It wasn’t just the logistics of implementing distributed scoring that was a challenge, the accompanying online training had to be designed in a way to anticipate questions the scorers might have and include the answers as part of the pre-packaged training.
An unanticipated benefit was that I was able to see the training —all of it— well ahead of scorers and it could be reviewed before the scorers received it. The trainings were built with the knowledge and it did not receive any slant or off the record opinions from trainers, it was obvious that it was more objectively presented and was free of trainer biases. Also, the fact that scorers could train and score at their own pace was such a relief for the teachers.
Another benefit of online training, (and I think anyone who has previously sat through a training session will vouch for this) is that scorers no longer had to sit and listen to another scorer who was having difficulty understanding or accepting scoring standards or rubrics. There was no longer that scorer in the room causing frustration for the rest of the team.
Cultivating Professional Development
We soon realized that allowing teachers to score at their own pace in a comfortable environment (warehouses are not that comfortable) resulted in an individualized opportunity for professional development.
Teachers could look at their quality statistics without the humiliation of a scoring director calling them aside to tell them they’re not doing well. The entire process of performance scoring became much less personal. We went from reading handwritten essays where students “dotted” their “i’s” with tiny hearts to being told “your scoring is not acceptable” via computer message. But that’s how it should be, isn’t it? Much more objective. One of the main goals has always been to take a very subjective process and make it as objective as possible. Objectivity is especially critical today at a time where students and teachers are working in virtual environments.
Setting the Stage for Next Fall
As we look at the challenges ahead, I think it’s really important to reflect on how far we have come, especially in the area of performance assessment. Next Fall, we are facing a school year unlike any other we’ve had before. Change is coming and based on my past experience we’ll look back one day and be able to say, “Can you believe that is how we used to do it”?
Daisy
Like what Daisy had to say? Check out some more of her reflections here, here, and here!
