Why the Fact-Check Became Its Own Genre and How to Check the Checkers

Imagine a referee who also owns the stadium, prints the programme, writes the rulebook and announces the score. Now imagine he is also, from time to time, a participant. You would want to watch him closely. You would certainly not hand him a sword and say, "Rule on what is true."

Folks, that is roughly the position of the modern fact-check, and I want to be fair about it before I am not fair about it. Checking facts is a good thing. Checking facts is, in fact, the oldest and most honourable part of the newspaper business. What changed over the last twenty years is that the checking turned into a brand, the brand grew a rating scale, the rating scale grew a color chart, and the color chart acquired the power to decide what millions of people on large platforms were allowed to see. Somewhere in that journey, a chore became a tribunal.

This post tells the story, explains how the ratings work, points out the structural problem, and ends with something useful: a method you can use to check the checkers. No sneering at fact-checking as such. Only an insistence that the people who grade others should be graded.

From the copy desk to the brand

Once upon a time, fact-checking was invisible. A newspaper or magazine employed researchers who verified quotes, dates and names before publication, and when they did their job well, nobody noticed. The check was an internal quality control, like a proofreader.

Then came a shift. In 2003 the Annenberg Public Policy Center at the University of Pennsylvania launched FactCheck.org, a project devoted to reviewing the accuracy of statements by political figures. In 2007 two more entered the field. PolitiFact, from what was then the St. Petersburg Times in Florida, began rating political claims on a scale, and the Washington Post started its own Fact Checker column, which in time became known for its "Pinocchios." PolitiFact won a Pulitzer Prize for national reporting in 2009 for its coverage of the 2008 campaign.

Notice what was new here. These were not internal checks on the paper's own work. They were external checks on the statements of politicians, published as standalone products with a name, a format and a rating. The fact-check had become a genre, with conventions as fixed as a sonnet: quote the claim, summarize the context, call experts, deliver a verdict.

Why did the genre take off? There are honest reasons. Political speech is full of dubious statistics and half-truths, and a reader benefits from someone running the numbers. There are also commercial reasons: a rating is shareable, a verdict is clickable, and a color-coded scale makes a good graphic. Nobody forwards a nuanced paragraph. Everybody forwards a "Pants on Fire."

How a rating scale works, and where the judgment sneaks in

Most fact-check outlets use a scale. PolitiFact's runs from "True" down to "Pants on Fire." The Post hands out one to four Pinocchios. Others use their own labels, such as "Mostly False" or "Misleading."

Here is the point to grasp. A scale turns a judgment into a measurement. And the scale is only as objective as the process that assigns the label.

Consider what must happen to assign "Half True." Someone has to decide which part of the claim matters. Someone has to decide how much context a fair-minded listener would have supplied. Someone has to decide whether a literally true statistic is being used to create a false impression, and how false. Someone has to decide how many deceptive elements make it "Mostly False" rather than "Half True." None of that is arithmetic. It is editorial judgment about meaning, framing and likelihood of misunderstanding, performed by a human being with a worldview.

Do not mistake this for a complaint that the judgment is wrong. Often it is perfectly reasonable. My complaint is that the format hides it. A rating arrives dressed as a measurement, with a number-like label and a cheerful graphic, and the reader cannot see the judgment calls that produced it. A good check will lay them out in the text. The rating on top still sends the signal that a verdict is the same as a fact.

The problem of checking opinions

The deepest difficulty is that political claims come in three kinds, and the genre blurs them.

1. Facts. The unemployment rate was a specific number on a specific date. A bill contains a specific provision. A court ruled a specific way. These are checkable, and a checker should check them.
2. Predictions. "This law will cost jobs." "That tax will raise a trillion dollars." These are claims about the future. They can be assessed against models and track records, but they cannot be called simply true or false, because the future has not happened.
3. Values and interpretations. "This policy is unfair." "That program is wasteful." "This reform helps working families." These turn on a standard of fairness or worth, and reasonable people differ about the standard.

A fact-check that treats a value claim as a fact claim is performing a category error. It is like grading a poem on its accuracy. And the temptation to do so is strong, because the genre rewards verdicts and a prediction or a value judgment resists them.

Here is a hypothetical, labelled as one. A politician says, "This bill will devastate small businesses." A checker examines economic studies, finds that estimates vary, finds some that point to modest effects, and rates the claim "Mostly False." But the word "devastate" is a value-laden prediction. A small-business owner whose margin is thin might experience the same law as devastating. The checker has taken a contested forecast and handed it a grade as if it were a statistic. That is not fact-checking. It is commentary in the costume of fact-checking.

The same hazard runs in every direction. Left-leaning claims about the effects of a minimum wage, right-leaning claims about the effects of a tax cut, any claim built on a forecast, will be vulnerable to being scored when it should be debated. A genre that cannot say "this is a disputed prediction, here is the range of views" and must instead choose a color has a design flaw.

When fact-checks became a gatekeeping system

For a while, all of this was just journalism, and you were free to ignore it. Then came the shift that turned a genre into infrastructure.

After the 2016 election, Facebook announced in December of that year a program to work with third-party fact-checking organizations. Stories flagged as false by those partners could be labelled and shown to fewer people. Other platforms adopted variations. The effect was to convert the fact-check from a commentary that readers could take or leave into an input to a distribution system. A rating could now reduce a post's reach and cost its publisher an audience.

That is a very different thing. A newspaper's opinion that a claim is false is speech. A platform's decision to throttle that claim based on a partner's rating is a power over what the public sees. And the ratings, as we have seen, mix verifiable facts with judgments about context. When the mix is fed into an enforcement system, the judgments carry the force of rules.

I will say plainly what I think about the private side of this. A private company is entitled to decide what it hosts and how it labels it, and I defend that right, for reasons I lay out in Moody v. NetChoice and the Unfinished Question of Who Owns the Feed and Publisher or Platform, Eight Myths About Big Tech and the Law. But a company that outsources its judgment to a handful of partners inherits their blind spots, and users are entitled to notice and to criticize. And when government officials press platforms to act on the checkers' calls, that is a different matter entirely, which I examine in Murthy v. Missouri Lost on Standing, Yet the Record It Built Still Stands and Misinformation Is Not a Legal Category, and Government Should Stop Pretending.

On January 7, 2025, Meta announced that it would end its third-party fact-checking program in the United States and move to a "community notes" model, in which users contribute context to posts, a system modeled on the one used on X. Whether that is better is an open question. Community notes have their own weaknesses, among them that they can be slow and that they depend on who chooses to participate. But the decision shows something worth noticing: even the company that launched this partnership program concluded the arrangement had problems; Meta's announcement said that experts, like everyone else, have their own biases and perspectives, and that those biases affected what got checked and how. The reader should neither gloat nor panic. A tool that its biggest customer stopped using deserves a skeptical look, and so does its replacement.

The steelman: why checkers deserve respect

I want to state the best case for fact-checkers, because it is stronger than their critics sometimes admit.

Politicians lie, exaggerate and mislead, all parties, all eras. A public that cannot catch them will be misled more often. Fact-checkers do the dull, necessary work of reading the report, calling the economist and finding the original number. Many of their checks are carefully sourced, link to the primary documents and treat claims from both parties with the same procedure. Some of them have rated claims by politicians on the right and left as false in roughly the same ways, and some checks have embarrassed politicians who deserved it. When a statement is plainly false, such as a fabricated statistic or a misquoted bill, a check is a public service.

And the fact-check is not the same as a censor. A fact-check is more speech, the remedy that our constitutional tradition prefers. Thomas Jefferson put the principle in his first inaugural address in 1801: "error of opinion may be tolerated where reason is left free to combat it." A fact-check that answers a claim with evidence is reason combating error. If it stopped there, I would applaud.

That is the steelman, and I accept most of it. Now the answer.

The rebuttal in five points

1. Selection is a choice, and it can lean. There are millions of political claims a day. The checker picks which ones to examine. A checker that examines claims from one side more often, or chooses the most vulnerable claims from one side and the most defensible from the other, will produce a pattern that looks like bias even if every individual check is accurate. The sample is the editorial decision, and the sample is rarely disclosed. Ask not only "Was this check right?" but "Why this claim, and why not the opposing claim from the same week?"
2. The rating conflates fact and judgment. As described above, a single label hides the weighing of context and intent. A reader should demand that the check separate what is verifiably wrong from what is arguably misleading.
3. Checkers check opinions as if they were facts. Predictions and value judgments are treated as claims that can be graded, which gives a false sense of precision.
4. The checkers are not themselves checked. A fact-checker that errs may or may not publish a correction, and the correction rarely travels as far as the original rating. Who fact-checks the fact-checkers? In a healthy system, other checkers, other newsrooms and informed readers, which is why a monopoly of checkers is more dangerous than a crowd.
5. When ratings drive enforcement, the stakes change. A rating that throttles distribution is a ruling with consequences, and a ruling should come with a process: notice, a chance to respond, a standard, a way to appeal. Most fact-check partnerships did not offer anything like that.

None of these points says fact-checking should stop. They say it should be held to the standard it applies to everyone else.

A method for checking the checker

Here is the practical part. When you meet a fact-check, run this test.

1. Read the claim as the speaker said it. Find the original statement, not the checker's paraphrase. Did the checker quote the whole sentence or a trimmed version? Many disputes dissolve or deepen at this step.
2. Identify the type of claim. Fact, prediction or value judgment? If it is a prediction or a value, treat any verdict as an opinion about the claim, and look for whether the check says so.
3. Look for the primary sources. Does the check link the actual data, the actual bill, the actual transcript? A check that cites only other news stories or unnamed experts has not done the whole job.
4. Separate the finding from the rating. Read the body of the check and write down in one sentence what was shown to be inaccurate. Then look at the rating. Does the rating match that sentence? A trivial error labelled with a severe rating is a sign that the label is doing editorial work.
5. Run the mirror test. Find a similar claim by someone on the other side and see how the same outlet handled it. If you can find a close parallel treated very differently, that is a data point. One parallel proves little. A pattern across many is worth noting.
6. Check for a correction. Has the check ever been updated or corrected? Is the update visible, or buried? Outlets that fix errors openly deserve more trust than outlets that never seem to make any.
7. Ask what you would conclude with no rating at all. Read the body, ignore the color, and decide for yourself. That is the habit that matters most.

If you apply this to a check you agree with, you will probably find it holds up, and that is satisfying. If you apply it to a check you disagree with, you may still find it holds up. Applying it both ways is what makes you a reader and not a partisan.

What this says about trust

Conservatives sometimes respond to fact-check bias with a blanket distrust of all checking, and I think that is a mistake. The problem is not that someone checks. The problem is a genre that has more authority than its method justifies. The answer is better method, more transparency and more competition, not the abolition of scrutiny.

Notice that conservatives and everyone else have a stake in this. Today's checker favours the claims you dislike. Tomorrow's may be staffed differently. A system built on the premise that a small group can be trusted to rule on truth for everyone is a system whose capture will be worth fighting over, and a system that is worth fighting over will be captured. The safer design is one in which no single arbiter exists, in which anybody can publish a check, anyone can check the check, and the reader makes the final call. That is slower than a color-coded verdict. It is also how free people get to the truth.

For more on how outlets can mislead without lying, see Seven Words Reporters Use to Tilt a Story Without Telling a Single Lie. For the sourcing standards that sit behind many stories, see Anonymous Sources, Who Gets Them and What a Reader Should Demand. And for a case where an official body tried to referee truth itself and lasted four months, see The Disinformation Governance Board Lasted Four Months, and That Was Too Long. The site's own media watch page collects tools for reading the press critically.

The case in brief

1. Fact-checking began as an internal newsroom job and became a branded genre: FactCheck.org in 2003, PolitiFact and the Washington Post Fact Checker in 2007.
2. Rating scales turn judgment into the appearance of measurement, hiding the weighing of context.
3. Facts can be checked. Predictions and values can only be argued, and treating them alike is a category error.
4. When platforms tied ratings to distribution from 2016, a commentary became a power, and Meta's January 2025 move to community notes shows the model's weaknesses.
5. The answer is more transparency, more checkers and a reader who tests every verdict with the seven-step method.

What you can do: pick a fact-check you disagree with, find the original statement and the primary sources, and run the seven steps before deciding who was right.