Benjamin Disraeli: “There are three types of lies: lies, damned lies, and statistics”

In the 1950s, when this book was written, Britain was considering changing over from the Imperial measurement system to the Metric System. Obviously that’s a pretty massive change, to shift from miles to kilometers, pounds to kilos, and from measuring things in ancient, archaic, out-dated ways to measuring in a more standard and modern way. At the time this was considered to be a huge shift (today there are only three countries stuck in the Imperial past that haven’t got with the times… Myanmar, Liberia, and world’s largest economy, America).

Gallup surveyed the Britons and found that 33% of people didn’t even KNOW what the metric system was, let alone how to use it or how it might become the country’s new standard for weights and measures. The Sunday newspaper was outraged at this statistic, so they conducted their own survey, and found that 98% of their readers DID know exactly what the metric system was and what its implications were for the country. How could two survey end up with a 16.5X result? One saying 33% were unaware, one saying only 2% were unaware. This is clearly not a small error, this is How To Lie With Statistics. 

The Gallup poll was conducted by a reputable organisation that does these kinds of surveys all day every day. They conducted a rigorous assessment of a broad cross section of the population. The Sunday paper had their own interests in mind, and conducted a different type of survey. In order to be included in the Sunday paper’s survey, firstly you had to obviously read the Sunday paper. Also, you had to cut out half of a page, answer the questions, then send it back. That takes a special type of person (apologies if you regularly respond to newspaper surveys… but if you do, you’re probably not listening to our podcast or reading our blog posts anyway). One argument the newspaper might make is that well-educated people read their publication, which is why they scored so high. Another argument though – if you DIDN’T know what the metric system was, why would you bother with the time and effort to snip out and survey and spend money to mail it in? If you had a big ego and didn’t want to admit that you didn’t know something, you’d probably go and research what the metric system was, then send in the survey saying “YES”. All of these things contribute to skewed data and dodgy stats. This is how to lie with statistics. It’s not surprising they ended up with completely different results. 

This is obviously a somewhat trivial example, but lies with statistics are more common than you realise. Any time you see a statistic, you should consider that it might be a lie. 

You need to learn some of these tricks so that you can defend yourself. The author says that the crooks know exactly how to manipulate data and statistics, so the honest people must learn the tricks as a matter of self defense. (Or, if you’re that way inclined and a little sneaky, you can sue some of these tricks to manipulate data of your own and lie with statistics a little more). 

In the podcast episode and in this blog post, we’re going to cover:

• Sampling Errors
• Biased Sampling
• The different types of “averages”
• How you can use and unrelated number to answer a different question
• And How you can fight back against these dodgy statisticians

BIASED SAMPLING

Let’s say you’ve got a barrel full of beans. Some are red, some are white. If you’re a pedantic person, like Ashto, you might want to know exactly how many beans of each colour you have. The only way to be completely certain about the exact figure is to top them all out and count them one-by-one. That’s a bit of a pain in the arse, and it’d take a pretty long time. So you want to find a short-cut, a way to get a good-enough estimate that you can be pretty confident in:

• You plunge your hand into the barrel and grab a bunch of beans. 
• You count how many in that handful are red and how many are white.
• You extrapolate these proportions to make an educated guess about the whole barrel (by estimating the number of handful there are in the barrel, you can multiply out the number of beans of each colour in your hand)

Of course, if all the whites are at the bottom and all the reds are at the top, then scraping a handful off the top isn’t presentative of the whole barrel. And if you’ve got tiny hands and can only pick up one or two beans, you’re not getting a good enough sample size. But if you’ve got a big enough handful, and that handful is ‘representative’ of the whole barrel, then you’ll get pretty close to an accurate estimation.

One extreme example of how to lie with statistics through sampling errors is a questionnaire asking people: “do you like answering questionnaires?”. The researcher found that almost 100% of people like to answer questionnaires!! What an incredible result!!?! But of course… the only people who are going to answer that questionnaire… are people who like answering questionnaires! Your sample has effectively eliminated anyone that might say “no” to this questionnaire, so your results are heavily biased. Similarly, psychiatrists might say that “pretty much everyone is crazy”, but remember who is in the sample of people going to see a psychiatrist (note that this book was written in the 1950s, some of these examples didn’t hold up that well over time…). Another one that doesn;t stand that test of more politically-correct times: a plastic surgeon saying that almost all women have weird, small boobs. 

SO… The results you get from a sampling study is only as good as the sample that it is based on. Unless the sample is truly “representative” of the population as a whole, from which every potential source of bias has been removed, then the study is effectively useless. This is NOT to say that a sample has to be an exact match of the entire population: it doesn’t have to be 50% men 50% women, the exact splits of religions, incomes, number of people in their family, level of education, etc etc etc. It doesn’t mean that every element of society is IN the study, just that everyone has an equal CHANCE to be a part of the random sample.

Cherry-Picking The Stats You Want

“If you can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing”. Wise words of advice from the author Darrell Huff. “In the daze that follows the collision of statistics with the human mind, hardly anybody will notice the difference. The semi attached figure is a device guaranteed to stand you in good stead – it always has”.

A fictional example (that feels all too real) is about a pharmaceutical company announcing that they’ve made a breakthrough. They’ve found a new medicine that cures the common cold! Of course, it’s very hard to prove that you’ve cured a cold… but there are plenty of other things you CAN prove to make it LOOK like you’ve cured a cold. You can publish a Lab report that shows half an ounce of  your active ingredient killed 31,108 germs in a test tube in under 11.5 seconds. While you’re at it, make sure the laboratory has a reputable name – pick a famous university or government organisation to conduct the lab study. Use a photo of  a doctor in a white lab coat holding up a test tube and looking closely at it in the light, putting this at the top of the report.

It’s not up to you to point out that killing artificially created germs in a test tube doesn’t necessarily mean that you can kill real germs in a human body. And you don’t have to say that the germs you killed are actually the germs that cause the common cold. And it doesn’t mean that because you did it once you can do it again. By whipping out your fancy stats, you can fool most people into believing your claims.

Or, you’re trying to market a new product. You announce that your “Juice Master 5000 extracts 26% more juice!”. But… more juice than WHAT? More than the best-performing competitor? More just than hitting it with a hammer? More juice that a toddler trying to use a hand-held juicer? Calling out statistics like this without comparison is a common way of how to lie with statistics. 

Fighting Back: The Opposite of How To Lie With Statistics

So far, we’ve talked about how people can manipulate data to make stats looks like whatever they want. If you want to fight back and see through some of these tricks, there are some questions you always need to ask. When you see a dodgy stat, ask:

• WHO SAYS SO? Work out who is telling you this, and what they’re trying to achieve. They’re sharing statistics that are favourable to them, so recognise that there is inherent bias there. Imagine the headline: “27% of doctors smoke Throaties Cigarettes”. If Throaties Cigarettes funded that research study, you might want to think twice about it’s accuracy and legitimacy. 
• HOW DO THEY KNOW? Look at their research method. If this doesn’t hold up, or if it sounds like it was loosely cobbled together, you might recognise some of the data’s flaws
• WHAT’S MISSING? If you say that you;ve got 40% greater customer satisfaction… greater than what? Something is missing from this statement. You’ve only given half of the statistic, perhaps the other half wasn’t favourable to your story.
• DID SOMEONE CHANGE THE SUBJECT? Like the common cold cure we discussed earlier – they said that their medicine killed germs, they didn;t say that it cured a cold. They’ve changed the subject, they’ve answered a different question
• DOES IT MAKE SENSE? Some times, you need to apply a bit of common sense. If someone is trying to pitch you their businesses and they say they’re growing at 300% every year and soon that have captured the whole market, it doesn;t necessarily make sense that these stats will continue forever. Similarly, a study funded by the prostate cancer research society found that there were 8m active cases of prostate cancer in the US. But in the male 50+ age group at the time, the high-risk group, there were only 7m men aged 50 years or older… Clearly, that one didn’t make sense. 

Hopefully this has given you a couple of extra tools for your tool belt. You can either use them to defend against the tricksters, or if you’re that way incline you can use these for yourself as a way of how to lie with statistics. 

If you’re keen for more books, sign up to our monthly email newsletter: whatyouwilllearn.com/email