Other films, not critics, pick ‘great’ movies

"We find that ultimately it is the creators, the filmmakers themselves, who will determine which movies are important, not the expert critics," says Luís Amaral. (Credit: JD Hancock/Flickr)

Filmmakers, not critics, could be the best way to gauge a movie’s significance, according to new research.

The study shows that the best predictor of a movie’s significance is how often a movie is referenced by other movies.

“Movie critics can be overconfident in spotting important works, and they have bias,” says Luís Amaral, the leader of the study and co-director the Northwestern Institute on Complex Systems. “Our method is as objective as it gets.”

He and his colleagues are the first to systematically compare different approaches for estimating a film’s significance. They considered metrics for measures both subjective (critical reviews, awards, public opinion) and objective (citations, box office sales).

The researchers found their automated method of movie citations is better at predicting greatness, especially in movies 25 years old or older, than these runners-up: the expertise of movie critics (a group of critics or a single critic), the wisdom of the crowd, the numbers of awards won, and the amount of box office sales, among others.

The research team conducted a big data study of 15,425 US-produced films listed in the Internet Movie Database (IMDb). Specifically, they looked to see how well an approach predicted a movie’s inclusion in the National Film Registry of the US Library of Congress, which is akin to the National Baseball Hall of Fame.

Star Wars, Psycho, Casablanca

In their analysis, the researchers found the number of times a movie 25 years or older is referenced by other movies best predicts inclusion in this registry of American films deemed “culturally, historically, or aesthetically significant.”

Films with the most long-gap citations that also are in the National Film Registry are The Wizard of Oz, Star Wars, Psycho, Casablanca, and Gone With the Wind.

“Directors keep coming back to movies that are significant,” Amaral says. “If you show a little bit from Psycho, such as referencing the shower scene, you are putting that whole movie in front of the viewer of the new movie.”

“There is something about a movie that is hidden to us, but there are measurable things, such as critic ratings, awards, and referencing by other filmmakers, that hint at this hidden element—a movie’s significance,” he says.

“We find that ultimately it is the creators, the filmmakers themselves, who will determine which movies are important, not the expert critics.”

Also important, the researchers write, is that the automated method can easily be applied to older films for which no other rating may be available.

Next: scientific papers

Amaral enjoys movies, but what he really wants to do is develop a method for identifying the most significant scientific papers. Given the varied sizes of scientific fields, the sheer number of citations is not sufficient for determining greatness, he says.


“More than 1 million scientific papers are published each year worldwide,” says Amaral, professor of chemical and biological engineering in Northwestern’s McCormick School of Engineering and Applied Science.

“It can be difficult to distinguish a good scientific paper from an average one, much like the movies. My next goal is to develop a good measure of scientific citations to get inside what is going on in the scientific literature.”

The US Army Research Office supported the work. The study appears online in the Proceedings of the National Academy of Sciences.

Source: Northwestern University