Separator
If Nobody Can Agree on Review Scores, Why Have Them at All?
Me
Thursday, September 23, 2010
EDITOR'S NOTEfrom Brett Bates

I give Dennis's analysis of review scores and why we don't need them a score of 1023 points.

I recently got tapped to write a review for Halo: Reach, and once again I find myself wishing that I could eschew scores altogether. Without some sort of formal agreement as to what the numbers mean, scores become inconsequential.

The worst offender is the website Metacritic, notorious lord and master of the review score. It can't quite figure out how to translate non-score review systems into numerical expressions. A letter grade of “A” is equal to 95 points in the American grading system, but Metacritic doesn't care. It's translated numerous "A" scores for Halo: Reach to an inflated 100 points.

Other review sites, such The Escapist and GamePro, use a five-star system, ostensibly to escape numerical scores and focus more attention on the qualitative. Unfortunately, this just winds up feeding the Metacritic inflation beast. Generally, a rating of five stars means that you'll probably like the film, restaurant, book, or video game receiving it, but it doesn’t mean that the object being reviewed is “perfect.” A score of 100 implies perfection, and Reach is far from perfect. Yet reviews for The Escapist and GamePro come up as 100 on Metacritic.

 

Review scores are subjective, obviously. So while criticizing the way Metacritic translates them is easy, it becomes more problematic to criticize the actual scores originally given in the reviews themselves. But there’s something fair to be said about the psychology review writers employ when selecting their scores. In order for the scores to have even any subjective meaning, they do need to be compared to bars set by other games.

This got me thinking about the eventual release of BioShock Infinite. When I look at its world of Columbia, the themes being investigated, and design aspects like the Skyline transportation system and power combinations, I see the potential for a game which could once again, like the first BioShock, become a benchmark for first-person shooters. I have no reason to think that Irrational Games is going to drop the ball on this title, so I feel secure in making an educated guess that the game will hit all the right marks from a critical perspective.

If that happens, how will critics score it? If the flawed but very good Halo: Reach nets 100 on Metacritic, what would be an appropriate score for Infinite? 110? When we’re dealing with game score systems that deal in 100-point scales, there’s absolutely no excuse not to show some discernment. Otherwise the score is bereft of meaning, which is why most intelligent consumers of the video-game media ultimately seem to shuck off the scores if they bother continuing to read reviews at all. Hey, If I didn’t have to write reviews myself, I’d be standing in the same crowd.

Of all the necessary evils of being a video-game journalist, review scoring is the one which smells the heaviest of sulphur.


Dennis Scimeca is the Editor in Chief of the website Game Kudos and a staff writer at Gamer Limit.. If you tweet him @DennisScimeca he will get right back to you, because he's officially bored with Halo: Reach.

 
4
BITMOB'S SPONSOR
Adsense-placeholder
Comments (11)
September 21, 2010


Personally, I have no problem with the concept of review scores. However, you always come across slightly biased and hype-wagon scores which then get thrown into the Metacritic average. The perfect score of 10 or 100% should never be issued, why? Simple, no game offers the perfect presentation values and gameplay then coupled with replay value, not one.


Awesome_center_redux_2
September 22, 2010


If the perfect score should never be issued Jim, why do we even have it? Why can we not use it? If we have a 10 point scale and can't issue a 10 because a game is never "perfect" than we're simply saying that 9 is a perfect score. It doesn't make any sense.



Read the review (the part that actually explains what a game is like), then decide if you like the look of the game. Possibly download a demo, decision made. We don't need any sort of scoring system in my mind.


Img950653
September 22, 2010


As I've gotten older and I've developed a more refined pallette for what I like in games, I've found I care less and less about reviews - particularly their scores. As a kid I can remember reading EGM and carefully pouring over reviews for games I had zero intention of ever playing. Somehow, that information seemed important to me at the time. Nowadays I could really care less if the AV Club gave Reach a B-, or if Game XYZ got a 3/5 on this site but a 4/5 on that one. It's not that I don't care about game journalists' opinions, because I listen to and read about their unfiltered thoughts extensively. But the realm of the game review seems so intellectually confining - much more so than book or movie reviews, and it feels like work to get through one about a game I care about. Chances are if I have it in my head that I'm going to purchase a game, a review isn't going to affect my opinion.


Mario_cap_avatar
September 23, 2010


Amen, dude.



I've been reviewing games for over 3 years and this issue still bugs me. I like assigning scores for games within my own reviews - to compare what me, myself, and I think of X game versus X game. I think on THAT level scores could be useful, but otherwise they're really just a cheap way for the reader to make a snap judgment on the general opinion of the review, and even then it's not accurate.



Since when did 7 become "meh" and 6 become "trash?" Last I checked, thsoe were both higher rather than lower...



Isn't a 1-10 scale pointless when it's really just a 6-10 scale?


Img_20100902_162803
September 23, 2010


Handling review scores and awards have their inherited issues, but it is the bread and butter of video-game critique (pretty much universal in movies, book, tv and so on). Of course the reviewer may have to wary with caution, is Halo Reach the best game of this year, when Medal of Honor, Call of Duty Black Ops, GT 5, etcetara, are yet to be released?


Jonathan
September 23, 2010


It's funny. I used to think that those numeric scores actually mean everything, but they don't. The scores don't capture the nuances of visual style and gameplay. And as much as I like Halo, pssshaw. I've been there before. I admit that I play Halo for the multiplayer fun, but it never gives me the surreal adrenaline rush of jumping off rooftops in Mirror's Edge. So an 100 rating is highly debatable.



 



Back in the old days, games were much easier to judge, because of the lack of genres and the limited graphics capabilities. Now that we have all this realistic environments with super-powerful graphics cards, we really are reaching a point where a numeric rating means even less than before. And all these ratings can't even match the precious feeling of finding rare retro games about ninjas fighting satanism.


Me
September 24, 2010


Just wondering: is the editorial preference on Bitmob not to italicize game names? That got changed in editing. I'll just stop doing it in my posts if that's the preference, and no worries. And how about the spelling of "video game?" Bill Kunkel recently opined on the subject that if one uses "video game" the Google search results are vastly improved. Never thought of that before, but if it's true...



 



Anyway. :)



 



@  Juan - I'm not sure that "bread and butter" is entirely true. I think of scores more as an option, not the standard, anymore. Too many A-list sites have eschewed scores in favor of letter grades or star systems.



 



Personally, I think Kotaku does it right. No scores altogether...but I'm made to understand that many sites refuse to abandon some sort of system that can be translated into a score by Metacritic because the traffic from MC is intense. However, G4TV asked MC to stop listing their scores because G4 didn't like the way MC was translating them. That's a ballsy move that I really appreciate. Only the highest-traffic sites can afford to do that, however...and it does make one wonder why IGN doesn't follow suit. I can't imagine they need the MC traffic, either.



 



@ Eddy - You'll find a lot of conversation online regarding that point. I think of it nowadays as "the 7-9 score system" as I'm pretty sure I've read a few editorials that called it that precisely. What it comes down, IMHO, is using scores that the reader will understand. On one of my sites, we actually got into a debate about whether to use the Eurogamer or more typical score systems. Some of our European writers were afraid that our scores would be top-heavy if we used the normal system.



 



To a point that's true, but in the end it comes down to said writers correcting for that and scoring what they think is appropriate, and the site tends to grade games lower than other sites, which I'm rather proud of, actually; but in the end we had to consider what most gamers understand the scores to mean, and set up our system accordingly.



 



We also went through four or five big-name sites that actually spell out their scores in terms of text definitions and compared them...and there was absolutely no rhyme or reason to any of it. A score of 7, if I remember correctly, was where we found the most concurrence between all the systems, but the rest of the scores could vary extremely greatly. Hence, to be frank, we stopped worrying about it so much in the face of direct evidence that scores are utterly meaningless. Yet we had to use them, because that's what the reader expects...and also because we want to get listed on Metacritic next year. :(



 



Like I said, sulphur. Deals with the devil. But that's the business we're in. What eases my conscience a little is that the mainstream media has transformed into the same sort of profit-minded (read: traffic-minded) enterprise, so it's not as though video game journalism is a lesser-than in terms of respectibility, if that makes sense.


Jayhenningsen
September 24, 2010


Dennis - That is correct. We do not italicize game names. What we do (during editing) is create a clickable "tag" link to the first mention of a particular game in an article. Subsequent mentions get no special formatting other than the appropriate capitalization.



We prefer "video game" as two words unless it's used as an adjective, and then it gets a hyphen.

i.e. -  "I enjoy video games, but I hate video-game journalists."


Bithead
September 24, 2010


It's interesting to note that most movie reviews work from, at the very most, a 5- or 4-star system.  Or something as simple as "thumbs up/thumbs down."  Most book reviews don't even have scores.  In fact, I can't think of one publication (other than Entertainment Weekly) that actually gives a numerical score or its equivalent to a book, even though they still review them extensively.  Perhaps video games will one day reach that level of critical acceptance. 



It's interesting to see how certain mags are changing their process. In newer issues of GamePro, (as many BitMobbers should know) they've gone to more of a Metacritic-type approach, filtering through many of the reviews already out there and giving a wider spectrum of opinion across the board, instead of one reviewer's idiosyncratic take.  I don't know if I like that better, necessarily.  In fact I know I don't.  But it's one way of dealing with both the print mag's problem of lead time, and the simple fact that one person's opinion may or may not convey the best overall picture of a game's reception.


Halo3_ce
September 24, 2010


I stopped reading reviews about a year ago. I now listen to about 10 podcasts a week while i'm doing busywork or going to sleep, so that's how I get my opinions about games and such. I highly recommend this because you don't have to deal with numbers BS, it's just the reviewer telling you what they think about the game. I've definitely become more open about the games I play since doing so. If I hear Dan, or Garnett Lee, or Jeff Gerstmann raving about something I'm much more likely to check it out than if I see their name next to a review score no matter how good it is. That said, I am constantly following the news in the industry, so I'm able to form my own opinions about most every game out there; but hearing these guys' opinions has definitely opened my eyes a little bit.


September 27, 2010


Scores are organic to the reviewer.  Each reviewer's scale is based on subjective material.  Cuz face it... Games aren't getting 10s because the controls were precise.  If you have a reviewer that your favor or one who's opinions and preferences are very familiar to you, scores can be useful for quick information.  Other than that, they aren't useful because you don't know the reviewer well enough for the score to have any comprehensive meaning.  You have nothing to base it against.   Read reviews for context and content.


You must log in to post a comment. Please register or Connect with Facebook if you do not have an account yet.