A Field Guide to Lies: Critical Thinking in the Information Age (13 page)

Things That Are Unknowable or Unverifiable

GIGO is a famous saying coined by early computer scientists: garbage in, garbage out. At the time, people would blindly put their trust into anything a computer output indicated because the output had the illusion of precision and certainty. If a statistic is composed of a series of poorly defined measures, guesses, misunderstandings, oversimplifications, mismeasurements, or flawed estimates, the resulting conclusion will be flawed.

Much of what we read should raise our suspicions. Ask yourself: Is it possible that someone can know this?
A newspaper reports the proportion of suicides committed by gay and lesbian teenagers. Any such statistic has to be meaningless, given the difficulties in knowing which deaths are suicides and which corpses belong to gay versus straight individuals. Similarly, the number of deaths from starvation in a remote area, or the number of people killed in a genocide during a civil war, should be suspect. This was borne out by the wildly divergent casualty estimates provided by observers during the Iraq-Afghanistan-U.S. conflict.

A magazine publisher boasts that the magazine has 2 million readers. How do they know? They don’t. They assume some proportion of every magazine sold is shared with others—what they call the “pass along” rate. They assume that every magazine bought by a library is read by a certain number of people. The same applies to books and e-books. Of course, this varies widely by title. Lots of people
bought
Stephen Hawking’s
A Brief History of Time
. Indeed, it’s said to be the most purchased and least finished book of the last thirty years. Few probably passed it along, because it looks impressive to have it sitting there in the living room. How many readers does a magazine or book have? How many listeners does a podcast have? We don’t know. We know how many were sold or downloaded, that is all (although recent developments with e-books will probably be changing that long-standing status quo).

The next time that you read that the average New Zealander flosses 4.36 times a week (a figure I just made up, but it may be as accurate as any estimate), ask yourself: How could anyone know such a thing? What data are they relying on? If there were hidden cameras in bathrooms, that would be one thing, but more likely, it’s people reporting to a survey taker, and only reporting what they remember—or want to believe is true, because we are always up against that.

P
ROBABILITIES

Did you believe me when I said few people
probably
passed along
A Brief History of Time
? I was using the term loosely, as many of us do, but the topic of mathematical probability confronts the very limits of what we can and cannot know about the world, stretching from the behavior of subatomic particles like quarks and bosons to the likelihood that the world will end in our lifetimes, from people playing a state lottery to trying to predict the weather (two endeavors that may have similar rates of success).

Probabilities allow us to quantify future events and are an important aid to rational decision making. Without them, we can become seduced by anecdotes and stories. You may have heard someone say something like “
I’m not going to wear my seat belt because I heard about a guy who died in a car crash
because
he was wearing one. He got trapped in the car and couldn’t get out. If he hadn’t been wearing the seat belt, he would have been okay.”

Well, yes, but we can’t look at just one or two stories. What are the relative risks? Although there are a few odd cases where the seat belt
cost
someone’s life, you’re far more likely to die when
not
wearing one. Probability helps us look at this quantitatively.

We use the word
probability
in different ways to mean different things. It’s easy to get swept away thinking that a person means one
thing when they mean another, and that confusion can cause us to draw the wrong conclusion.

One kind of probability—
classic probability—
is based on
the idea of symmetry and equal likelihood: A die has six sides, a coin has two sides, a roulette wheel has thirty-eight slots (in the United States; thirty-seven slots in Europe). If there is no manufacturing defect or tampering that favors one outcome over another, each outcome is equally likely. So the probability of rolling any particular number on a die is one out of six, of getting heads on a coin toss is one out of two, of getting any particular slot on the roulette wheel is one out of thirty-seven or thirty-eight.

Classic probability is restricted to these kinds of well-defined objects. In the classic case, we know the parameters of the system and thus can calculate the probabilities for the events each system will generate. A second kind of probability arises because in daily life we often want to know something about the likelihood of other events occurring, such as the probability that a drug will work on a patient, or that consumers will prefer one beer to another. In this second case, we need to estimate the parameters of the system because we don’t know what those parameters are.

To determine this second kind of probability, we make observations or conduct experiments and count the number of times we get the outcome we want. These are called
frequentist
probabilities. We administer a drug to a group of patients and count how many people get better—that’s an experiment, and the probability of the drug working is simply the proportion of people for whom it worked (based on the
frequency
of the desired outcome).
If we run the experiment on a large number of people, the results will be close to the true probability, just like public-opinion polling.

Both classic and frequentist probabilities deal with recurring,
replicable events and the proportion of the time that you can expect to obtain a particular outcome under substantially the same conditions. (Some hard-liner probabilists contend they have to be
identical
conditions, but I think this takes it too far because in the limit, the universe is never
exactly
the same, due to chance variations.) When you conduct a public-opinion poll by interviewing people at random, you’re in effect asking them under identical conditions, even if you ask some today and some tomorrow—provided that some big event that might change their minds didn’t occur in between.
When a court witness testifies about the probability of a suspect’s DNA matching the DNA found on a revolver, she is using
frequentist
probability, because she’s essentially counting the number of DNA fragments that match versus the number that don’t. Drawing a card from a deck, finding a defective widget on an assembly line, asking people if they like their brand of coffee are all examples of classic or frequentist probabilities that are recurring, replicable events (the card is classic, the widget and coffee are frequentist).

A third kind of probability differs from these first two because it’s not obtained from an experiment or a replicable event—rather, it expresses an opinion or degree of belief about how likely a particular event is to occur. This is called
subjective probability
(one type of this is Bayesian probability, after the eighteenth-century statistician Thomas Bayes). When a friend tells you that there’s a 50 percent chance that she’s going to attend Michael and Julie’s party this weekend, she’s using Bayesian probability, expressing a strength of belief that she’ll go. What will the unemployment rate be next year? We can’t use the frequentist method because we can’t consider next year’s unemployment as a set of observations taken under identical or even similar conditions.

Let’s think through an example. When a TV weather reporter says that there is a 30 percent chance of rain tomorrow, she didn’t conduct experiments on a bunch of identical days with identical conditions (if such a thing even exists) and then count the outcomes. The 30 percent number expresses her degree of belief (on a scale of one to a hundred) that it will rain, and is meant to inform you about whether you want to go to the trouble of grabbing your galoshes and umbrella.

If the weather reporter is well calibrated, it will rain on exactly 30 percent of the days for which she says there is a 30 percent chance of rain. If it rains on 60 percent of those days, she’s underestimated by a large amount. The issue of calibration is relevant only with subjective probabilities.

By the way, getting back to your friend who said there is a 50 percent chance she’ll attend a party, a mistake that many non–critical thinkers make is in assuming that if there are two possibilities, they must be equally likely. Cognitive psychologists Amos Tversky and Daniel Kahneman described parties and other scenarios to people in an experiment. At a particular party, for example, people might be told that 70 percent of the guests are writers and 30 percent are engineers. If you bump into someone with a tattoo of Shakespeare, you might correctly assume that person to be one of the writers; if you bump into someone wearing a Maxwell’s equations T-shirt, you might correctly assume that they are one of the engineers. But what if you bump into someone at random in the party and you’ve got nothing to go on—no Shakespeare tattoo, no math T-shirt—what’s the probability that this person is an engineer?
In Tversky and Kahneman’s experiments, people tended to say, “Fifty-fifty,” apparently confusing the two possible outcomes with two equally likely outcomes.

Subjective probability is the only kind of probability that we have
at our disposal in practical situations in which there is no experiment, no symmetry equation. When a judge instructs the jury to return a verdict if the “preponderance of evidence” points toward the defendant’s guilt, this is a subjective probability—each juror needs to decide for themselves whether a preponderance has been reached, weighing the evidence according to their own (and possibly idiosyncratic, not objective) internal standards and beliefs.

When a bookmaker lays odds for a horse race, he is using subjective probability—while it might be informed by data on the horses’ track records, health, and the jockeys’ history, there is no natural symmetry (meaning it’s not a classic probability) and there is no experiment being conducted (meaning it’s not a frequentist probability). The same is true for baseball and other sporting events. A bookie might say that the Royals have an 80 percent chance of winning their next game, but he’s not using probability in a mathematical sense; this is just a way he—and we—use language to give the patina of numerical precision. The bookie can’t turn back the hands of time and watch the Royals play the same game again and again, counting how many times they win it. He might well have crunched numbers or used a computer to inform his estimate, but at the end of the day, the number is just a guess, an indication of his degree of confidence in his prediction.
A telltale piece of evidence that this is subjective is that different pundits come up with different answers.

Subjective probabilities are all around us and most of us don’t even realize it—we encounter them in newspapers, in the boardroom, and in sports bars. The probability that a rogue nation will set off an atomic bomb in the next twelve months, that interest rates will go up next year, that Italy will win the World Cup, or that soldiers will take a particular hill are all subjective, not frequentist:
They are one-time, nonreplicable events. And the reputations of pundits and forecasters depend on their accuracy.

Combining Probabilities

One of the most important rules in probability is the multiplication rule. If two events are independent—that is, if the outcome of one does not influence the outcome of the other—you obtain the probability of
both
of them happening by multiplying the two probabilities together. The probability of getting heads on a coin toss is one half (because there are only two equally likely possibilities: heads and tails). The probability of getting a heart when drawing from a deck of cards is one-quarter (because there are only four equally likely possibilities: hearts, diamonds, clubs, and spades). If you toss a coin and draw a card, the probability of getting both heads and a heart is calculated by multiplying the two individual probabilities together: ½ × ¼ = ⅛. This is called a joint probability.

You can satisfy yourself that this is true by listing all possible cases and then counting how many times you get the desired outcome:

I’m ignoring the very rare occasions on which you toss the coin and it lands exactly on its side, or it gets carried off by a seagull while it’s in midair, or you have a trick deck of cards with all clubs.

We can similarly ask about the joint probability of three events: getting heads on a coin toss, drawing a heart from a deck of cards, and the next person you meet having the same birthday as you (the probability of that is roughly 1 out of 365.24—although births cluster a bit and some birthdates are more common than others, this is a reasonable approximation).

You may have visited websites where you are asked a series of multiple-choice questions, such as “Which of the following five streets have you lived on?” and “Which of the following five credit cards do you have?” These sites are trying to authenticate you, to be sure that you are who they think you are. They’re using the multiplication rule. If you answer six of these questions in a row, each with a probability of only one in five (.2) that you’ll get it right, the chances of you getting them right by simply guessing are only .2 × .2 × .2 × .2 × .2 × .2, or .000064—that’s about 6 chances in 100,000. Not as strict as what you find in DNA courtroom testimony, but not bad. (If you’re wondering why they don’t just ask you a bunch of short-answer, fill-in questions, where you have to provide the entire answer yourself, instead of using multiple choice, it’s because there are too many variants of correct answers. Do you refer to your credit card as being with Chase, Chase Bank, or JPMorgan Chase? Did you live on North Sycamore Street, N. Sycamore Street, or N. Sycamore St.? You get the idea.)