There are some good ideas for fixing the terrible, arbitrary scale games are reviewed on. But the best idea might be to just get rid of them.

SimCity
SimCity EA

The gaming website Polygon took some minor heat this week when it retroactively updated its review of the newly released SimCity. Polygon takes a unique route toward grading games: the site makes it clear in its policy that a review can be altered when something about the game changes. It exercised that policy by lowering SimCity's score from a 9.5 to an 8, then again to a 4, when server issues prevented the reviewer (and a lot of other players) from being able to play the game.

Here's a snippet of the policy:

Polygon's reviews and database have been built based on the idea of updates, or "bumps," as I've called them. If a game changes in a substantive way, we can add an update to our reviews that informs you how and why, and we can modify our scores accordingly. This will appear on the reviews in question as a timeline of that game's evolution and our corresponding recommendation (or lack thereof). The original review score will never vanish or go away, but our readers will be able to better understand where our opinions as a site reside over time for games we review.

That's a great policy! Games are, for better or worse, more changeable in a world where you can download new content, or experience new issues, as a game ages. That puts them in a completely different category than movies or music, which, for the most part, are static throughout their lives. (Maybe Highway 61 Revisited gets an outtake or remastering, but it's not fundamentally changing.) Technology does not affect your enjoyment of a movie or an album in the same way; nobody has ever given a movie a bad review because the projector screwed up.

On the surface, there's reason to think scaled game reviews are helpful to consumers. An abstract opinion gets turned into digits, and those digits can be objectively compared. Can't decide what to drop your $60 on? This system makes it easy. Art designed for consumption is difficult to quantify, and inherently encourages rating.

Trouble is, the system in the game review world, most popularly some variation of a 10-point scale with increments, is especially distorted. Take a look at this graph, via Joystiq:

IGN And GameSpot Reviews
IGN And GameSpot Reviews:  Metafuture

It shows a sampling of reviews from the gaming websites IGN and GameSpot back in 2006, and you can see something similar today by taking a gander at GameRankings.com. Reviews tend to lean toward the higher end of their scales, closer to a seven or eight than a five average. (That could mean something devious, but I'm more inclined to believe it's a natural function of people who like a lot of games reviewing those games.) That doesn't matter if someone is trying to compare two games--they're both on that distorted scale, after all--but if someone is making a decision to either buy or not buy a game, and isn't familiar with the distortion in the ratings system, they might have some trouble. You're reading a review and decide that eight, after all, is pretty good. Certainly above average. But what you might end up with is a middling game.

There are ways to solve this problem. You can turn a review, as Kotaku has, into a binary answer to the question: Should you play this game? (Maybe not totally binary: SimCity got a "Not Yet" from them.) You can get rid of reviews entirely. You could move the scale down to five and hope that mitigates the distortion. You can audit your reviews and see if they fall into a bell curve.

But the best and most honorable way to fix the issue to do away with scales entirely. Give the reader some credit: not everyone is going to skim down to the bottom, see what score a game got, and move on, and why should reviewers cater to that type of reader, anyway? Each person decides what to play based on personal tastes, and they should have to read a review to get the necessary context for their decision. In fact, I'd wager, gamers are more likely than other review-readers to weigh their options by reading the text of a review. A game, after all, can be a big investment in time and cash--at $60 for new home console games, it's six times as much as a movie ticket or an album on iTunes--and that necessitates more information. You can still add and update to a review as time goes on and the games change, and because text (hopefully) has nuance that numbers can only guess at, it means any updates will seem less dramatic but more informative.

So what does a digit put down on a scale offer? At best, it's a reflection of what's in the review, which makes it repetitive. At worst, a distorted scale changes the reader's opinion of the text, which nobody writing a review wants. Reviewers don't spend all that time testing games, composing their thoughts, and creating a critical essay to have their work replaced with a number.

20 Comments

Yes, I was so worried about this problem of game scoring.

LoL

Game reviews are extremely important... Why would I spend $60 on something that sucks? What other product do you spend that much money on and have no option of getting a refund if you're dissatisfied?

I personally like Conan O'Brian's scales. "On a scale from Kirsten Stewart to Jimmy Stewart, I give this game a Steve Buscemi."

ppardee, I don't think you understood or even read this article.

The problem here is that it's being assumed the ratings are distorted and that 5 should be the average.

What's not being acknowledged here is that the average game is actually pretty damn good and that game makers, over the course of 30+ years, have become quite good at making games...on average.

The scale is not distorted. The understanding of gaming, games and the gaming industry is distorted...

The way I see it... most gamers think that any game below a 70% on Metacritic is not worth their time, and certainly not below a 60%. In all honesty... I don't have much time to play games that don't average in the 80s or 90s.

And guess what? That's how we get rated in school. Anything below a 60 is failing. So, I have no problem with the scoring.

GameSpot used to be better, and IGN accepts bribes. They are like senators of the gaming industry.

@Vega_Obscura = If every game is "Above Average" then that is the Average and therefore should be a 5/6 on a scale of 1-10.

The game review system is broken because you can't trust the sources, plain and simple. If they wanted a "good" review system they would break down the reviews into categories and rate then on a 3 point scale of Below/Average/Above.

Graphics = Average for games released at this point in time
Sound = Average for games released at this point in time
Controls = Average usability vs games released at this time
Functionality = Game is playable with few noticeable bugs
so on and so forth with other catagories.

Then "number" those a -1, 0, and +1 and add the results. Any game scoring a 0 or better is your "Average" game and should be acceptable to most players. Anything less then 0 and expect complaints. But that kind of review system would make sense.. so no one will use it.

"...The game review system is broken because you can't trust the sources,..." ABSOLUTELY!

A critical rating/grading distinction should be made; and it is clear that neither the author of the article nor many of the people commenting are familiar with it: NORM REFERENCED evaluations compare the thing to its "peers" and result in a normal or bell shaped distribution of scores vs. CRITERIA based evaluations which assess how the thing meets certain specified criteria (without taking into consideration the other things that have been previously evaluated). Most schools used to use norm referenced grading (only the top kids can get an A and some always fail), whereas many schools now use criteria based grading (getting an A is proof of mastery of x,y,z criteria, and an individual's grade is not affected by who is in your class). A meaningful discussion (and article) would be about whether or not game ratings should be norm or criteria based; otherwise you are just muddying the waters and getting nowhere...

To further clarify the issue, current game ratings are not "distorted" as the author claims; they are simply not norm referenced. If something is not norm referenced then please don't expect the average to be in the middle of the rating scale because it likely won't. Vega (post 4) is saying that game reviews are criteria referenced (and most meet those criteria) which is why most scores are high (which I think is mostly true, but maybe or maybe not good; I'm still thinking about that).

What companies need to do is RELEASE MORE DEMOS!

I rarely if ever buy games unless it's from a trusted company and an obvious buy because of the history I've had with the franchise.

And when I do finally say "ok I'll buy it." It's usually from a used store so it's cheaper than the 60$ price tag, because again I don't want to spend 60$ on a game that may not be worth it.

I would probably end up buying more games if there were more demos... But then perhaps developers are helping me specifically because I can save money for other things instead.

I have been a serious gamer for 20 years now. I like having an overall numerical score for a game, here's why:

First, it is not always readily apparent from a review what a reviewer's overall score is. More than a few times I have surprised at the score given a game after reading the review. Sometimes a reviewer rants about a game's flaw, but then give it a 90. Other times a reviewer uses the kid gloves on a game that ended up with a 60. So, a review's tone may not convey the reviewer overall opinion of a game. A number, on the other hand, is hard to misinterpret.

Second, I usually ignore reviews for games in the genres I'm not interested in. I'm not going to read hundreds of reviews in a genera where I may play 1 game in 5 years. However, if a game has an unusually high score, I might read the review to see if it is worth giving a chance.

Scores don't replace reviews, they enhance them. If you abolish their use, you are going to lose readers. That is one less tool we can use to make our decision. I am not going to frequent a game site that does not use some sort of rating system.

This is something I've been considering over the past couple of days. I do like to see numerical scores when I read videogame reviews, but something has recently complicated this: SimCity.

How does one rate SimCity? Maxis created a great game, but the experience was absolutely thrashed by their publisher's distribution requirements and always-on digital rights management, which required a permanent connection to the internet. The idea, itself, is anti-consumer, but then you throw in the fact that the servers were not stable and far too few, and you have a major problem.

The game is outstanding, but its publisher's greed (Electronic Arts, for the record) has sabotaged the experience. This is why we're in a bind with the reviews. Gaming sites have averaged around a 70% rating for the game, while consumers have rated it at a staggeringly awful 16% because of all the garbage EA forced on them.

Hard to say whether the developer, Maxis, should be punished by poor reviews when all of the game's flaws can be attributed to the puppet master, EA.

So what should the score be? 1.6, or 7.1?

The answer is much simpler.

No one ranks the shovel-ware that Disney/Pixar movies, Barbie, etc pump out on an annual basis. Why? Because it is garbage riding on its brand appeal. Add all those 1's into the pile and I bet your scale average comes down closer to 5.

Furthermore, this isn't random stats. These products have already been tested before release to (hopefully) assure a level of quality. It's like buying 2 iPods and assuming one of them will be broken before you open it, just because.

Yay for academics making commentary without practical thought.

Let's be clear, EA is trash and ruins games with stupid decisions. I stopped buying EA games after they screwed up Swtor.

Let me say that Bioware created a great game, Swtor, but EA came along and screwed it all up.

I like the idea for games to offer a DEMO verson of the game to tray, prior to buying.

GREAT IDEA!

Studies show that games that have demos sell fewer units than games that don't. That is why they have fallen out of favor in the games industry. They are more interested in selling games than making satisfied customers.

Also, the always online requirement for SimCity isn't for DRM purposes. The game itself was designed to crunch numbers on the SimCity servers. So there is no way to disable the always online requirement without redesigning the game.

Now, I'm not saying that EA doesn't love excessive DRM implementations, but you can't blame the outage on the EA. Maxis designed the game to require connection to their servers, and did not have enough servers to carry the load.

until I saw the receipt for $5631, I did not believe ...that...my best friend was like they say actualey bringing in money parttime from there computar.. there sisters roommate had bean doing this 4 only about 16 months and a short time ago repayed the loans on their appartment and bourt a brand new Maserati. read more at, http://www.jump14.com

SimCity EA . I think it's very attractive.The photo above has appealed to me. http://www.kizi66.com

Popular Tags

Regular Features


140 years of Popular Science at your fingertips.



Popular Science+ For iPad

Each issue has been completely reimagined for your iPad. See our amazing new vision for magazines that goes far beyond the printed page



Download Our App

Stay up to date on the latest news of the future of science and technology from your iPhone or Android phone with full articles, images and offline viewing



Follow Us On Twitter

Featuring every article from the magazine and website, plus links from around the Web. Also see our PopSci DIY feed


April 2013: How It Works

For our annual How It Works issue, we break down everything from the massive Falcon Heavy rocket to a tiny DNA sequencer that connects to a USB port. We also take a look at an ambitious plan for faster-than-light travel and dive into the billion-dollar science of dog food.

Plus the latest Legos, Cadillac's plug-in hybrid, a tractor built for the apocalypse, and more.


Online Content Director: Suzanne LaBarre | Email
Senior Editor: Paul Adams | Email
Associate Editor: Dan Nosowitz | Email
Assistant Editor: Colin Lecher | Email
Assistant Editor:Rose Pastore | Email

Contributing Writers:
Rebecca Boyle | Email
Kelsey D. Atherton | Email
Francie Diep | Email
Shaunacy Ferro | Email

circ-top-header.gif
circ-cover.gif
bmxmag-ps