Over the years, reviewers have wondered how our scoring system works. They’re often interested in finding out how 20 multiple-choice questions convert to a max score of 70. What follows is a little history, a little math, and more than a little lingering confusion.
Normal Scoring
Our scoring used to be done via Excel spreadsheets. And that basic architecture carried over when we integrated reviewers' scoring and our tracking via scoring forms.
Possible RAW scores for every question range from zero to 1 in quarter-point increments.
On our scoring forms, a 5 (worst possible score) is converted to 0, a 4 = 0.25, 3 to .5, etc., topping out where a 1 = 1
Although each of our 20 multiple-choice questions starts with a min of 0 and a max of 1, the TOP combined score is not 20, nor is it 100. It's 70.
Depending on the genre, there are different weighting factors applied to each section.
Content questions are weighted heavier than visual, for example, EXCEPT for children's picture books (for obvious reasons)
For most books, the order of weighting in descending importance is: content, tech, style, visual
The actual weighting factor numbers by genre are buried inside our spreadsheets. It all happens automatically.
Bottom line: you'll never be able to know exactly how each change in scoring will impact the final score -- none of us do. That's why we first ask reviewers to "score blind" (answering each question as objectively as possible, without knowing how their answers translate to scores or medals).
Reviewer Conference Scoring
Your second chance at scoring only happens after a reviewer conference. At this point, rather than requiring all reviewers to do a complete do-over score, we ask them to make changes using the scoring scales found in that reviewer conference form. Although any reviewer can opt to do another scoring form, since it's easier to complete the reviewer conference form, most choose that less time-consuming option.