The Influence of Metacritic on Game Sales

Written by Rob Savillo

Editor's note: How much of an effect does Metacritic's aggregate scores have on games? I've always wondered about this, but I've never analyzed the data. Luckily, Rob has, and he's sharing his examination of Metacritic scores and sales for May 2009 with us. -Jason

The September 2009 issue of Game Informer published a feature discussing the merits of review aggregator Metacritic through the lens of Glen Schofield, former general manager for Visceral Games, which developed and released Dead Space late last year.

Central to Schofield’s complaint is a lone score of 65 that caused the Xbox 360 version of Dead Space’s aggregate Metacritic score to drop one point from 90 to 89. Schofield states that “the difference between an 89 and a 90 is a big-ass deal.”

Games journalist Mitch Krpata criticized the article, claiming that Game Informer was calling for reviewers to inflate scores in order to keep the Metacritic aggregate high. Whether or not Game Informer actually implied such is irrelevant to the observation that review score inflation would be at the detriment of the gaming public.

However, we all know that publishers use Metacritic scores to pressure developers by tying those scores to monetary compensation. Are publishers justified in using Metacritic in such a way? I decided to investigate the issue by tracking game sales and their accompanying Metacritic scores to find out the relationship between the two.

In order to have a manageable sample, I limited my study to games released during the month of May 2009. I did this because my source for game releases and sales data, VGChartz, only publishes the first 10 weeks of sales. May is the first month that all of the titles released had sales data for the full 10-week period.

I also feel that 10 weeks of sales data is enough to draw conclusions between the possible effects of Metacritic scores on game sales. The only systems that I included are the Nintendo DS, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360, and Wii. Games needed to have sales data available in order to be included; in all, 31 games are in this study.

The above chart's representative of the study's full results. I’ve also highlighted several titles in order to give a sense of where certain games stand in the data.

We can see that game sales are only tentatively related to Metacritic scores. I say this because even if a game receives a high aggregate score, there's still the possibility of low sales numbers.

UFC 2009: Undisputed is the clear outlier here, with more than 1.6 million in sales and an aggregate score of 84. EA Sports Active and X-Men: Origins -- Wolverin e are the only other high-selling games -- the former’s Metacritic score is 81, and the latter’s score is 63.

The lowest scoring game is Leisure Suit Larry: Box Office Bust, which got an 18. It's also one of the worst-selling games in the study, moving just over 30,000 units. The data also show that there are no poorly scoring games that also sold well (exceptionally or not).

From this chart, we can see that there's a clear upward slope of games with higher Metacritic scores being more likely to sell more games. But we can also see that many games with a high Metacritic score also sell poorly.

Many of the games studied sold fewer than 200,000 copies, so let’s zoom in on that data.

Boom Blox: Bash Party received an aggregate score of 86, yet only sold approximately 60,000 units. In contrast, Night at the Museum: Battle of the Smithsonian received a score of 52 but also doubled the sales of Boom Blox: Bash Party.

Again, we see several outliers, such as Sacred 2: Fallen Angel, which sold more than 190,000 units with a Metacritic score of 71. We also still see the same upward slope as the full data chart, just less pronounced.

The data also suggest that the majority of game releases sell less than 60,000 copies regardless of their Metacritic score, though a majority of the lowest-selling games had a Metacritic score above 60 (a total of 15 games).

Of the 10 games that sold over 100,000 copies, four of them were based on films (X-Men: Origins -- Wolverine, Up, Night at the Museum, and Terminator: Salvation), two were sports related (UFC and EA Sports Active), and the rest were either remakes (Bionic Commando and Punch-Out!!), sequels (Sacred 2), or exclusives (Infamous). This suggests, at least to me, that there are more powerful factors that influence game sales than Metacritic scores.

Are publishers correct to use Metacritic scores as a primary factor in understanding how well a game sells? The data does suggest that no poorly scored games are going to also sell well; however, the data also suggest that a high Metacritic score isn’t a guarantee of high sales, either.

Additionally, most games appear to fall into the top left corner of moderate-to-high Metacritic scores and relatively low (less than 60,000) unit sales. While the profits from such sales are nothing to sneeze at, it’s generally not enough for games with budgets in the tens of millions.

My opinion from the data is that Metacritic scores are not as important as publishers would like to believe them to be. Other factors have a bigger influence on game sales. The data from this study's limited in providing answers to what those other factors may be and how much influence they may or may not have on game sales.

Methodology: Game release dates and sales data were collected from VGChartz. Only games that had sales data for the 10 weeks were included, with the exception of three titles (Raiden Fighters Aces, Magicians Quest: Mysterious Times, and Challenge Me: Brain Puzzles. For the purposes of this study, these games were assumed to have had zero sales for weeks nine and 10. All three games sold less than 25,000 copies combined.) All sales data used is based only on U.S. sales.

I combined cross-platform game sales and averaged Metacritic scores between platforms to arrive at one average Metacritic score for all versions of a game. Nintendo DS, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360, and Wii were the only platforms represented because sales data for other platforms was unavailable.

I used May 2009 as a sample in order to keep the data collection manageable. May was also the first month that all games released had 10 weeks of sales data available. A more thorough investigation could be completed by compiling all data from the first six months of 2009. Such an undertaking may be the subject of a follow-up study.

Comments (9)

As some one with an interest in video games and a BA in sociology, well done! Methodology and everything, nice Just be prepared for some "TLDR"

Anyway, Metacritic is such a joke for so many reasons, this article clearly shows one of them.

Andrew Gregory , August 27, 2009

I had a feeling that Boom Blox didn't sell well. At least I can console myself with the fact that it outsold Leisure Suit Larry!

Brett Bates , August 28, 2009

It seems that even though the practice of basing someone's salary upon a metacrtic score sounds shady, it is fair because in the end people are getting rewarded for making a good game. If publishers used sales to determine how much to pay developers they would be worse off.

Jason Monmouth , September 01, 2009

-1

Reviews very rarely influence sales, in any medium. I don't understand why so many people care what Metacritic says.

Ryan Higgins , September 02, 2009

Man, I think Bitmob needs to change the conversation. Truly. I can appreciate the issue, but it's come up a lot here lately. All folks can hope for in the case of a site like Metacritic is that there is a series of checks and balances when deciding which sites merit having their review scores posted.

I do suspect the same, though, in that the Metacritic score isn't the be all/end all of how well a game will perform. If it was, everyone would know the names Okami and ICO. For me, I see three main factors that seem to greatly determine the success of a game: 1.) The appeal to Walmart moms, 2.) The un-hyped appeal to hardcore gamers, and 3.) The occasional fluke.

Tony Capri , September 02, 2009

Wether this subject is irrelevant or not, this is a really well done project. I frickin' LOVE graphs and charts, for you cannot argue about them, only think and discuss them.

For instance, if I were a Game Publisher I would take one look at those graphs and think: "Okay, I need to make a game with some boom and market it enormously. It will sell 200,000 on marketing alone, and who cares if its good."

But fortunately, I am not a publisher - those bastards are even worse than I would be!

Lance Darnell , September 02, 2009

The sad truth is that metacritic is designed primarily for lazy/stupid people incapable of understanding what makes a game good and what makes a game bad so they stick to a simpler system of numbers.

I feel for developers having to put up with the bs number rating system overall, not just metacritic. If I were 'rating' games I'd go with 3 simple categories; good(you should play), bad(you should not play) and subjective(you may or may not like it but its for the most part worth playing). Hmm think I might right an article on this.

Aaron Betts , September 02, 2009

A well written article that basically shows how little influence Metacritic and reviews for that matter have on the overall sales of games.

Personally, I'm not a fan of reviews that attach a numerical value to something. For one, the weight of a numerical value differs between publications and even within that publication the reviewer attaches a different weight to it. Game Informer is pretty terrible when it comes to overall reviews. They generally blow smoke up developers' asses on poor to mediocre games and fellate the ones they give good scores too. Just about the only publications that I've seen to give constant and fair reviews with numbers attached are EGM, Next Generation and Edge magazine. I'd just like to see the emphasise on numerical scores removed and replace it with something simpler like a 'Buy' 'Rent/Borrow' 'Trash' sort of system. Or listing the pros and cons of a game. Sites like GameCritics, Kotaku and Crispy Gamer are ones that I'm a fan of when it comes to the way games are reviewed.

Travis Bolek , September 02, 2009

This is probably a reasonably good study if what you want is a measure of the correlation between metacritic score and sales numbers.
But this is not what is implied in either the editors note or in several parts of the article. Instead we are told that we are uncovering if metacritic affects sales.
Unfortunately there is absolutely nothing in this study that would be able to determine this.
The biggest problem is of course the lack of controlling for other factors when it comes to what drives sales. The article mentions this but instead of drawing the correct conclusion that nothing can be said about the effect of metacritic on game sales the eroneous conclusion is drawn that metactitic has low or no effect on game sales. This even though there are absolutely no evidence to support this statement.

Oscar Haglund , September 06, 2009