‘Miss Representation’ and The Importance of Good Statistics

September 17, 2014

In my many years of writing articles and occasionally fact-checking news media reports (see, for example, much of my book Media Mythmakers: How Journalists, Activists, and Advertisers Mislead Us), I’m continually baffled by the defensive stance that people often take when you fact-check their statistics. People often assume that if you’re checking their facts that you must be against them or what they’re trying to do.

In fact it’s exactly the opposite: I’m against misinformation and bad statistics–whatever the source, and especially when being offered in service of causes I support. I want to correct false data and bad logic because I assume that detractors will use any mistakes or misinformation to dismiss the whole idea as bogus. Sloppy research is often a sign of sloppy analysis and faulty claims. One or two typos and transcription errors can happen to anyone, but when there’s a pattern of misinformation it does more harm than good.

I appreciate it when on rare occasions people catch factual mistakes I make in my writing (obviously the sooner the better, because if my name is on it I want it to be credible), and I assume that other people are the same way. Yet often they don’t. As skeptics know, or should know, asking for evidence and checking the validity of statistics–and especially statistics that contradict common sense–is merely good critical thinking skill. Nonetheless, I’ve often been attacked and criticized for having the temerity to fact-check numbers and statistics offered by a wide variety of people, from feminist Eve Ensler to the researchers at eHarmony (see my article in the next issue of Skeptical Inquirer magazine)

I thought about this recently when I happened to come across the web site for a social justice film I reviewed a few years ago. In the 2011 documentary Miss Representation, writer/director Jennifer Siebel Newsom takes a look at the media’s influence on girls and women. She interviews a variety of people including Condoleezza Rice, Nancy Pelosi, Katie Couric, Rachel Maddow, and Gloria Steinem about the messages that mass media send. The main theme is that the media harms our culture, in part by telling girls that their value lies not in their brains but instead their beauty. What, Newsom asks, is this doing to American women? It’s an important and legitimate question, and one which the film offers a long list of bold facts and statistics about.

According to the film’s web site, “The film accumulates startling facts and statistics that leave audiences shaken, armed with a new perspective.” Unfortunately, little of that is true. The film is littered with factual errors and logical lapses. For example the director throws dozens of statistics on the screen on a wide variety of topics, all of them allegedly related to the topic (usually referencing gender bias, media consumption, etc.). One of the highlighted statistics that caught my eye read, “65% of American women and girls have an eating disorder” (it also appeared on their web site).

As someone who has researched and written about eating disorders for many years, I recognized immediately that the statement is not correct. It is simply not true that two-thirds of American women and girls have eating disorders, and certainly not calorie-restricting eating disorders such as anorexia nervosa or bulimia (those rates are closer to between 1 and 3 percent). I saw some other dubious statistics as well, but it would take days to fact-check all of them.

Why did this film, which cast itself as an authoritative resource (and even has a place on its web site for teachers to order the film for school), have misinformation in it? Didn’t someone check the data before they included it in the film? Any eating disorder specialist (or even a seven-second search on Google) would have revealed it could not be correct.  It made me wonder what other mistakes and errors the film contained. How can we expect to teach people media literacy if the experts and their materials have such disregard for accuracy?

Miss Representation has some important points to make–the discussion of gender bias in news reporting was pretty accurate, for example, and it briefly touched on problems of media conglomerates and deregulation. But when the film focused on the entertainment media–which was the majority of the film–its important points are obscured by exaggeration and illogical leaps.

I discussed some of these bogus eating disorder statistics in my Masters thesis (titled Misinformation in Eating Disorder Communications: Implications for Science Communication Policyavailable HERE), and wrote about it in a few places.

I recently discovered that the web page titled “The Sources for the Statistics” on the web site for the Miss Representation film has been taken down. I’m sure I’m not the only person to have questioned the film’s many obvious factual errors (even a fawning interview with Newsom and profile on the film with Mother Jones uncovered mistakes in Miss Representation‘s statistics; see the corrected addendum at the bottom of this article), and there may be some other plausible explanation for why the page giving the sources for their statistics–and only that page, as far as I can tell–cannot be found. But it’s likely that the filmmakers finally acknowledged that their educational and empowering documentary was also spreading misinformation.

Yet instead of taking the page down, why didn’t they simply fix the factual errors? After all, if it’s only one or two, it shouldn’t take more than a few minutes to correct the information and update the page. My guess is that it’s far more than one or two but perhaps dozens, and correcting them would obligate them (as filmmakers of an educational film marketed to teachers and schools) to fact-check all of their hundreds of statistics, which they would be loathe to do, and which would be deeply embarrassing for them since the original bogus facts and statistics remain in the film. It would hardly look good to be forced to provide an errata document for teachers and viewers listing all the factual mistakes in the film and correcting the information with accurate references. I’m guessing it’s easier to just take down the “Sources for the Statistics” page and hope no one notices.

It’s a good example of why it’s important to get your facts right in the first place, and why fact-checking numbers and statistics is not a hostile, debunking action but instead a helpful one–whether the person offering them recognizes it or not.