UnSubject has an excellent post on Metacritic. I strongly recommend reading it.
The most interesting thing about Metacritic these days--and what makes it more controversial than all the other review aggregators for other media--is that some game publishers are tying bonuses to the game's Metacritic score.
This seems a little odd to me. UnSubject's explanation is that royalties are tied to Metacritic because “it’s a quantitative measure of game quality, popularity and helps forecast sales”. I don't think this is the full explanation.
For one thing, Metacritic scores are a weak proxy for what publishers really want to learn: how much money did this game make? But publishers don't need a proxy to figure out how much money a game made, they have direct access to the sales figures. It would be more sensible to base bonuses directly off gross revenue, rather than an indirect measurement such as Metacritic.
The only theory I have is that gross revenue or sales can be thought of as a function of both game quality and marketing (and the sheer fickleness of the audience). However, marketing is traditionally the responsibility of the publisher. It seems unfair to pay or withhold bonuses if the publisher did a good or bad job on the marketing.
In theory, reviewers should not be affected by marketing. Metacritic scores should filter out marketing's contribution to gross revenue, and only represent the developer's contribution.
Of course, the fact that Metacritic tends to correlate well with popularity and sales generally means that this extra indirection is unnecessary. It would be interesting to see what games are outliers. Games where the Metacritic score did not predict sales or revenue; either a poorly-rated game selling many units, or a highly-rated game selling fewer units.
From the perspective of an ex-QA software person, they should tie bonuses to the number of bugs they get after release. You'd be surprised at how airtight software would be if that were the case....
ReplyDeleteThe problem with tying bonuses to income is that there are many, MANY games that are critical successes but commercial failures. It basically rewards developers for doing the "safe" thing, which is why we see annualized games that do little more then put a new campaign and a fresh coat of paint on last year's version.
ReplyDeleteThe problem with tying bonuses to Metacritic is that, as you say, Metacritic is pretty fickle, because the reviewing industry is pretty fickle. And that's not even getting into situations like Obsidian getting screwed out of a bonus from New Vegas because they got 84% instead of 85% (http://www.joystiq.com/2012/03/15/obsidian-missed-fallout-new-vegas-metacritic-bonus-by-one-point/).
And the problem with the scheme in general is that it's nothing but the publishers gaming the contracts so that they can keep more of the money they didn't do anything to earn.
That's my point about outliers.
ReplyDeleteUnSubject's analysis says that Metacritic is a good predictor of sales. In that case, there's really no point to tying the bonus to Metacritic instead of sales.
In the movie industry, criticism is NOT a good predictor of sales. So in that case, tying the bonus to the critic's ratings would be measuring something different.
Anything from Blizzard will likely score badly on Metacritic and sell tons of units. I'm not being cynical here although it might sound like I am.
ReplyDeleteIf you want to look at some examples of "outliers", here's a few (sales numbers taken from VGChartz, which means they ignore digital sales):
ReplyDelete- Fire Emblem: Awakening (92% Metacritic, <1 mil sales. Though Nintendo considered 300k a complete success a few months ago.)
- CoD: Black Ops 2 (74%~83% Metacritic, ~24.6 mil sales)
- Enslaved: Odyssey to the West (~80% Metacritic, ~1 mil sales)
- XCOM (~90% Metacritic, ~1 mil sales)
- Fallout: New Vegas (~84% Metacritic, ~6.5 mil sales)
- Spec Ops: The Line (~77% Metacritic, ~800k sales. Both of these results are criminal, as if this was a movie it would probably be the perfect example of a critical success, commercial failure.)
Really, if you look at a list of the top games, in either sales or reviews, you tend to see the other figure for them varying wildly. I would thus argue that it's not so much a cast of outliers, but that quality and revenue is straight up not linked at all.