What is noise?

Wherever you look at human judgements, you are likely to find noises. In real-world decisions, the amount of noise is often scandalously high. 

Medicine is noisy. Faced with the same patient, different doctors make different judgments about the diagnosis. 

Forecasts are noisy. Professional forecasters offer highly variable predictions. Whether it’s about the popularity of a new product, the change in the unemployment rate, the likelihood of bankruptcy of a particular company, or the results of the next election. 

Asylum decisions are noisy. Whether an asylum seeker will be admitted to the United States is like a lottery of which judge you get – some judges granted 5% of cases while others granted 88% of cases. 

Not only do humans disagree with each other, but they also disagree with themselves. For example, software developers were asked the projected hours it would take to complete a project. When they were surveyed again later that week, the answers differed by 71% from what they said a few days before. 

To improve the quality of our judgments, we need to not only overcome bias but also overcome noise. 

Judges & Judgements

Say someone has been convicted of a crime – shoplifting, possession of heroin, assault, or armed robbery. What should the sentence be?

Ideally, the answer shouldn’t depend on the particular judge that is assigned to the case. It would be outrageous if three similar people – convicted of the same crime – received three different sentences. Probation for one, two years for one, and ten years for the other. Unfortunately, we see this type of thing happening all over the world every day, and it plagues many organisations. 

It’s almost like a random lottery system. Who you meet within an organisation can drastically affect your outcomes. This random lottery determines which doctor you see in a hospital, which lawyer you see in court, which patent examiner assesses your application, and which trader manages your investment portfolio. By now, hopefully, you’re seeing how much of a problem noise can be; and more importantly, how ubiquitous it is. The noise is a consequence of the informal nature of the judgement, so, the conclusion is simple. Wherever there is judgement, there is noise.

Components of noise

Definition of judgement 

 A judgement is a measurement you make with your mind. Like measuring how long a line is by using a ruler, you’re measuring a situation using your brain. Some examples include:

• How long a jail sentence should be
• The chances that the mysterious blob on the MRI is a benign tumour or a malignant one
• Which job applicant is the best fit for the job
• The value of a stock and whether it is underpriced or overpriced. 

And now we’re going to break down noise into its component parts:

• Level Noise
• Pattern Noise
• Occasion Noise

Level Noise

 As any defence lawyer will tell you, each judge has their own reputation – the harsh ones are called “hanging judges”, while the more compassionate ones are called “bleeding heart judges”. Differences in their decisions reflect the judges’ characteristics – their backgrounds, life experiences, political opinions, biases, and so on. Researchers also found that judges with conservative ideology would be more likely to assign heavier sentences. But none of this has to do with the circumstances of the case or the defendant. It is more of a reflection of the judges’ personality trait. 

Pattern Noise 

After Level Noise (variation between different judges), you also have Pattern Noise (variation within a judge). 

Let’s use pattern noise to measure the variability between an individual’s judgements across time. That variability reflects a complex pattern in the attitudes of judges to particular cases. One judge may be harsher in general (level noise) but goes soft on white-collar crime (pattern noise). Another might be easier in general but goes extra hard on repeat offenders. A third might be near the average but goes easy when the accused is merely an accomplice and goes hard when it’s an old person because they should’ve known better.

Some responses to this might be unknown. The defendant might have reminded him of a particular hateful criminal, or who perhaps looked like his daughter. Whatever their origin is, these patterns are not mere chances. We would expect them to occur if they saw the same case again.

Occasion noise

It is less easy to observe the variability of our minds. Of course, we have all had the experience of changing our minds, even without learning new information. The film that made us laugh out loud last night now seems mediocre and forgettable. The person we judged harshly yesterday now seems to deserve our forgiveness. An argument that we had not liked or understood finally sinks in and now appears essential. We usually associate changes with relatively minor and largely subjective matters. 

In reality, our opinions do change without apparent reason. When wine experts at a major US wine competition tasted the same wines twice, they scored only 18% of the wines identically. The wine tasters tasting the same wine (without being told they were the same) thought that they were different.

You’re not the same person at all times.

Better Decisions

 Now that we’ve fleshed out what noise is, there’s an obvious question: How can we reduce noise? How can an organisation improve the judgements its professionals make? In particular, how can an organisation reduce judgement noise?

The 6 principles

 1.  The goal of judgement is accuracy, not an individual expression

Personal values, individuality, and creativity are essential in many phases of decision making. But when it comes to making a judgement about options, the expression of individuality is considered noise. Judgement can be improved by using algorithms and the adoption of approaches that make decisions less dependent on the idiosyncrasies of one professional. 

2. Think statistically and take the outside view of the case

 Use unique views for a unique perspective. 

 3.  Break down judgements into several independent tasks

Be clear on what you’re measuring. Divide complex judgement into multiple fact-based assessments and ensure that each one is independent of others. And consider minimising communication between the people making the assessments. 

4. Resist premature intuitions

Relying too much on first impressions can lead you down a single chute. Instead, take the following approach:

• Assume that your initial estimate is off the mark
• Think about a few reasons why that could be. Which assumptions have you made that could be wrong?
• What do these new considerations imply? Was the first estimate too high to too low? In which direction should you shift?
• Based on this new perspective, make a second alternative estimate. 

5. Obtain independent judgements from multiple judges then consider aggregating those judgements

The requirement of independence is routinely violated in the procedures of organisations – notably in business meetings, where the participants’ opinions are shaped by others. Because of the cascade effect, group discussions are likely to increase noise. The simple procedure of collecting participant’s judgements before the discussion should reveal the extent of noise and facilitate a constructive resolution of differences. 

 6. Favour relative judgements and relative sales

Measuring something with a relative perspective makes it less noisy than an absolute statement. Saying “this wine tastes better than this wine” is less noisy than saying “this wine is 6/10 and this wine is 8/10”. Our ability to categorise objects on a scale is limited, but our ability to make pairwise comparisons is much better.