Is Review Aggregation Worth Fixing?

Sharing is caring!

Open Critic discusses its approach

The strength and value found in game reviews come from their independence, their voice, the nuance conceived in grammar, not math.

Stripping away those voices, boiling down criticism to arithmetic and then aggregating that byproduct to find a mathematical average doesn’t just undermine a critic’s elucidations, it weaponizes them.

The practice of review aggregation has long held power over video games in one form or another. There was a time when many teams involved in the creation and marketing of a video game received bonuses based on scores. A point in either direction could mean a bountiful cash prize or the cancellation of a sequel.

While it seems that studios have moved away to some degree from this over-reliance, review scores and the sites that aggregate them still hold vast power.

Take the news this week of a bad actor creating hundreds of fake, free accounts to review bomb video game Kunai on Metacritic, apparently on a whim.

Late last year, I set about trying to interview the people in charge of the most popular review aggregating sites, but quickly found not everyone was open to answering questions.

Metacritic, for its part, pointed me to a series of links about how the site operates. GameRankings has since been rolled into MetaCritic, and Rotten Tomatoes gave up on including video games on its site back in 2008 to redouble its efforts on movies.

That leaves OpenCritic, an increasingly popular aggregator that launched in 2015.

Matthew Enthoven tells me that the site aggregates about 2,000 games a year and that while readership “fluctuates wildly” they have about 150,000 people on their site a month.

Open Critic’s approach breaks reviewers down into two categories: critics and top critics.

“There are 197 top critics, of which 190 are active,” Enthoven said. “These are sites that we proactively aggregate and include on OpenCritic. They’re also the ones included in the top critic average.

“There are currently 301 critics. These are sites that are not included in the top critic average but are included in the percent recommended statistic. These sites also submit their review metadata directly to us using the Portal.”

For a while, he said, the site was averaging two to three new critics per week, but recent work to overhaul their back-end meant new applications were temporarily suspended. Once a site is accepted, it is used for all games it reviews. To be removed from Open Critic, the site either has to request it or Open Critic has to determine — after a “thorough investigation” — that the editorial integrity of the publication has been compromised. That’s what happened with Brash Games in 2017.

While Open Critic does have top critics and critics, they don’t weight reviews based on a site’s standing, influence or any other factors.

“We believe that, fundamentally, reviews are written by individual people and are sharing an individual’s perspective,” Enthoven said. “That’s why we put such a high emphasis on critics. We aggregate the author that wrote the review in addition to the publication, and we have author pages that show any author’s review history with us.

“With that in mind, we don’t think it’s fair to say that one person’s perspective and experience is somehow more valuable than someone else’s.”

Finally, he said, the site does include non-scoring reviews, listed separately for a game.

“They are not included in the top critic average,” he said. “If the review contains a clear verdict (such as Eurogamer, GameXplain, ACG, etc.), then we will include them in the percent recommended score.”

While Open Critic’s approach to aggregation seems a bit more nuanced than Metacritic’s, they both still boil a creative work down to a grade, something that the people behind Kunai still don’t agree with.

“It’s weird to grade a piece of artwork with a score,” head of marketing’s Arnaud De Sousa told GamesIndustry.Biz. “We’re not making homework. We’re making a video game.”