What PolitiFact Didn’t Say

[Geee]

National Review:

The headlines were inevitable: “Fact-Checking Site Finds Fox News Only Tells the Truth 18 Percent of the Time,” “Analysis: Over Half of ALL Statements Made On Fox News Are False,” “Fox News wins battle for most-false cable network.” One particularly dim-witted account, written by Jameson Parker, reads: “A new analysis by PunditFact found that of every statement made by a Fox News host or guest, over half of them were flat-out false. What’s more, only a measly 8 percent could be considered completely ‘true.’ In other words, a fancy review of hundreds of hours of video confirmed what many who watch Fox News with any regularity already know: Fox News lies. A lot. Like all the time.”

 

The “study” from PunditFact (a subdivision of PolitiFact), which is not really a study, says no such thing. You do not have to rely on my word for that: PunditFact itself warns against using its figures “to draw broad conclusions,” e.g. that Fox News lies “like all the time.” (Like, is this like a 1980s Valley-girl movie, or are you just, like, functionally illiterate?) That is because the study is an exercise in drawing nonsensical conclusions from arbitrary data.

 

The most obvious problem — though certainly not the only problem, not even close — is selection bias: PolitiFact is a readership-driven online publication, and thus it exercises a great deal of discretion about which statements it chooses to evaluate and why. The most obvious factor is that it evaluates only statements that are disputed. Specifically, it evaluates only statements that are disputed and that its editors believe will be of some interest or benefit to its readers.