Separator
Redefining Numbers
Default_picture
Monday, February 22, 2010

This isn’t a rant about rating games with numbers. Conversations of that nature require a steady hand, an even temper, and a measure of restraint that I’m not capable of. Listeners of Bullshot! know that my expertise lies in beating the correct answer into your head as violently as it needs to be for you to comprehend it. This is a rant about defining numbers, and I’m about to commence the violence.

Most outlets, both in print and online, use a number system to rate their games. A lot of those outlets have chosen to use a 1-10 scale as their number system of choice. Not content with using their scale in the traditional manner, most of these outlets have also chosen to re-weigh the scale. Motivations vary, and I am not about to delve into any of them here. Instead I am going to attack the practice itself because it is maddening.

 

Numbers are a universal system. If I traveled to China and held a rock in each hand, everyone would know I was holding two rocks. Language is irrelevant, the concept of “two” is universal. If I were on a planet orbiting Alpha Centauri and had the same payload, the natives would understand the concept of two. They would also know that five more identical payloads would be ten, and halving those payloads would result in five rocks. On a scale 1-10, the Alpha Centaurians would know that five was in the middle, regardless of what they called it.

And that is the problem. The publications that insist on telling us that seven is in the middle of the scale are literally trying to redefine numbers. Numbers are constant, this isn’t up for debate. You can say “Haha, three is now average,” but that just means you are wrong.

Yes, you read that right: You are wrong. This isn’t a matter of personal opinion or preference; this is science, literally science. You cannot argue the worth of the number five because it will always be worth five. If you are grading things on a scale from 1-10, five will always be in the middle. Numbers less than five will always be below average, and numbers greater than five will be above average. I know it’s getting redundant in here, but the more I repeat it, the greater chance you have of absorbing it.

To put it in different terms, have a look at the color spectrum below. On the left is yellow, and on the right is red. As the colors blend into each other, they eventually become orange in the middle of the spectrum. When a publication tries to tell you that 7/10 is their average, it’s like they are telling you that orange is actually red. Well, sort of red. Oven roasted tomato to be precise. Doesn’t matter, it’s not orange. Orange is in the middle; it is the average of yellow and red.

Of course, the most common rebuttal to this argument is, “We don’t use a 1-10 scale,” which is confusing. Are we just leasing the numbers individually? If so, why lease all of them? Why not just lease the ones you intend to use? Seems wasteful to get the whole scale when you don’t intend to use it. Listen, if you are grading things where “1” is the worst and “10” is the best, then you are using a 1-10 scale. Why do I even need to explain that? And if you are using a 1-10 scale, the middle is fiv...ok, you must understand by now.

Now the question is: What is the harm in repurposing numbers? The answer is the lack of a unified collection of data. Alex talked about this recently on Pixelosophy, but I’m going to take it to its inevitable end. Sites like Metacritic and Game Rankings pull your numerical rating and throw it into a pool with a lot of other sites. They boil the pool down until they have an average score, and then they pin a badge on the game with that score. When a site like Game Informer gives a game a seven (average for their scale) and a site like Eurogamer gives it a five (average for their scale), the game’s mean score becomes a six. You now have a score that is not only skewed because of the data, but it also isn’t an accurate representation of either publication from which the initial scores were taken. Does that strike you as being as stupid as it should? If not, read it again until you understand.

But what can we do about it? We could all sign an online petition, boycott publications, or just sit around and get real angry about it. Or, we could even get real crazy and open up a dialogue with the people who make these decisions. There isn’t really a good answer here, but the first step is realizing that it is a potential problem. Hell, maybe I’m wrong, and maybe people like seven being the new five. I think you are crazy and a scientific heretic, but that’s just one man’s opinion. I know that the tone of this article wasn’t the most pleasant, but sometimes that’s what you have to do. Over the years I’ve learned two very important things. First, never wipe towards Mecca. Second, if you want to get someone’s attention, you have to preface your statement with “Hey Motherfucker.” I hope that this got your attention.

Brendan is one of the co-founders of fourplayercoop.com. While his editorial musings are few and far between, he can be heard every week as the color commentator for Four Player Co-Op's Bullshot! podcast.

 
2
BITMOB'S SPONSOR
Adsense-placeholder
Comments (7)
Demian_-_bitmobbio
February 22, 2010
Hey Jesus - would you mind registering with your real name via the 'account settings' link in the top right? You could always put "TheJesusFish" in quotes between your first and last name if you just wouldn't feel right without it.
Default_picture
February 22, 2010
Sorry, old habits die hard.
Demian_-_bitmobbio
February 22, 2010
Thanks! Hmm not quite sure why you're showing up as 'a guest,' I'm hoping that'll be fixed with the site upgrade coming within the next day or so.
Default_picture
February 27, 2010


Hey B.J. - I enjoyed your article.  I've felt the same way for quite some time now. This rating system is too perverted to ever allow any number range to work in the future. However, if I'm not mistaken, the letter grading system (A-F) is known everywhere and would make an excellent replacement.  The letters already have feelings associated with them: A is a must play, B is a good game with some flaws that could be overlooked, C is average; D is poor but still playable, and F is avoid at all costs.



Sure, there's still the problem of having a grading system at all, but I feel this would be a step in the right directions.  At the very least, it wouldn't hurt the heads of all us math oriented folks.



Any chance we could make it happen?


Default_picture
February 27, 2010


Hey BJ, it's good to see I'm not the only one who feels this way about rating system used in games writing. I try to avoid pimping my blog and drawing the attention away from bitmob, but you may find my blog post on a similar topic interesting: http://www.oldschoolhard.com/2010/02/gaming-writing-inhale-the-fail/


Default_picture
March 01, 2010


B.J. - Unfortunately, I have three bones to pick with your article, and they're all math related.



1) "They would also know that five more identical payloads would be ten..." - That would be 12 rocks. If you have 2+(5*2)=12. The problem here is the word 'more.' That and the wording is kind of confusing as to whether or not you mean to include the original two rocks.



2) You're misinterpreting 'average' or 'mean' as median - the middle number. Either way, you're wrong. The median, and average of a 1-10 scale is actually 5.5, not 5. If it was a 0-10 scale you'd be more right.



3) Having '7' as the average score of a site's ratings just means that the sum of their scores awarded divided by the total number of awarded scores is 7. What if a site included all their reviews in a database and adjusted the score each game got based on all the reviews!? (Hint: Insanity)



It's like saying exams are out of 100 points. If five people take the exam and score 100, 100, 90, 70, 40 then their average is an 80, regardless of what the scale for scores being used is. Average isn't 50 just because that's the middle number on the scale.



Metacritic and Gamerankings might be better if they used all the reviews from a site to compute the site's average score and then ranked each game based on that, but then you'd get into the whole sampling error since no site reviews every game to be released, and generally only reviews games that either the editors are interested in or that the editors think their readership will be interested in. Maybe that's why 7 is average, the sites are trying to correct for not reviewing every crappy game that would score a 1 or 2 on their scales. Maybe not.



So, overall, you're wrong. I get what you're trying to say though... kinda. :)


Default_picture
March 02, 2010


Sorry for my lack of commenting; I was on a Mass Effect 2 bender, I'm sure you understand.



Anywho, where to start...



Evan and Dan, thanks for the kind words. Dan, I read your article while humming "we shall overcome". That is either creepy or...no I guess just creepy.



Jordan, I understand that this is the internet and that we all think we are smarter than everyone else here, but come on man, you are nickel and diming me.



When I wrote the article, I wrote it with some assumed knowledge built into it. That included the knowledge that most 1-10 scales are actually 0-10 scales; that end of the scale being reserved for the absolute dregs. But, I didn't specify that in an article that could be interpreted as an attack, so I will grant you that point. On a true 1-10 scale, 5.5 is the average. You tomaytoed my tomahtoe. It doesn't change the fact that in any uniform distribution (like, say, a 0-10 scale), the median and the mean are the same.



As for the confusing payload quote, again, guilty as charged your honor. Most people understood that as 10, but I worded it poorly enough that you could exploit it.



Your last point, however, is a lot of assumption and theory without a lot of evidence. Yes, in a perfect world (or bell curve), 7 could very well be the average. But I would bet that averaging the scores of most publications that use 7 as the made up average wouldn't result in 7 being the real average. I think the real villain here is what Evan talked about: the fact that, in school, 70% was average. But that's a different article for a more patient writer (that means you Dan).



As for your last point, in the immortal words of Ron Burgandy, "Agree to disagree".


You must log in to post a comment. Please register or Connect with Facebook if you do not have an account yet.