Why Are People Bad At Probability Calculations?

Introduction

In Behavioral Economics, there are several examples of the mistakes people make, some of the flaws in their patterns of thought and reasoning, and numerous biases they may have when it comes to addressing lab experiments and even real-life situations.

However, these aformentioned mechanisms can always be further expanded when understanding what the roots of these thought patterns are. Despite most mistakes people make and the biases they hold can be explained through past experiments and even evolutionary reasoning, some still loom around and are yet to be answered.

Today, as young curious individuals ourselves, we decided to tackle one of the most common mistakes people make when it comes to probability calculations and rational reasoning.

Some of the questions we are trying to answer are:

• Why do people miscalculate probabilities when answering questions such as the “test with 99% accuracy” question?
• Why do they think their answer to the question is the correct one?
• How do people tend to approach the question?
• Are people inherently bad at mathematics or logical reasoning?
• Can we explain the root cause of these mistakes they make?

Before elaborating on the results of our experiment and how they did or did not match with our hypotheses, we would like to briefly remark upon how we chose to conduct our experiment and who the participants of our experiment were.

The Experiment

The experiment was conducted among 39 people in total. 37 of our participants being current high school students and 2 of them being adults.

Above, you may see the grade distributions of our high school student participants which they will be attending the next academic year.

We conducted our experiment via the utilization of a free surveying service, Google Forms. The survey we used consisted of 3 questions in total.

These questions being:

1. What grade will you be studying in next year?
2. You go to see a doctor. The doctor performs a test with 99% reliability. This means, 99% of people who are sick test positive and 99% of the healthy people test negative. The doctor knows that only 0.1% of the people in the country are sick. The question is: if you test positive, what are the chances you are sick?
3. (optional) Please briefly explain why you answered as such:

The first 2 questions asked were multiple-choice questions and the 3rd a free-response question. The possible answers for the first question were the 5 different grade levels that the participants might be attending next year.

The possible answers for the 2nd question were (in order of appearance from top to bottom):

1. 100%
2. 99%
3. 9%
4. 1%
5. 0%

The only information provided to our participants in regards to what we were doing and why we were conducting this experiment before answering the survey questions (i.e. asking them to fill out a survey) was: “The form is completely anonymous. The responses will be used in a summer school project on Behavioral Economics.” No further information or insight into the experiment and what in actuality it was assessing was provided. However, detailed information was provided after an individual participant finished the survey in cases where they asked what the correct answer of the 2nd question was.

Correct Answer of the 2nd Question & Bayes Theorem

By utilizing Bayes’ Theorem, we can clearly see that the correct answer to the 2nd question is around 9%. However, were our participants able to find the correct answer as well?

Presentation of Hypotheses & Participant Expectations

To put it simply, our central hypothesis in regards to how the answers to the experiment will be was that a large number of participants would not be able to answer the question correctly.

Nonetheless, knowing that prior experiments already showed the same result, we decided to provide several hypotheses as to why our participants might have failed to find the correct answer which would hopefully help us answer some or all of the initial questions we proposed.

Our main hypotheses as to why they failed to answer correctly include:

• Our participants were simply bad at mathematics and probability calculations.
• As the test had a 99% rate of accuracy, they thought how prevalent it is in the population was redundant in the probability evaluation.
• Despite having mathematics and probability calculation knowledge, our participants weren’t aware of Bayes’ Theorem thus failed to choose the correct answer.
• Our participants were arguably young, thus, it might have caused their logical reasoning to not be as good as adults.

When looking at the data from our experiment and the participant profiles, we focused on these aforementioned hypotheses and tried to see if they were in actuality, true or false.

Results of the Experiment

Out of all 37 high school student participants who answered the questions, a whopping 89.2% (33 participants) answered the question incorrectly.

Above, you may see the answers given to the survey questions by our high-school student participants.

78.4% of the participants (29 participants) answered 99%, whilst 10.8% of the participants (4 participants) answered with 1%. The percentage of participants who answered correctly with the answer of 9% was only 10.8%, corresponding to 4 participants out of a participant pool of 37.

After seeing that our participants had, as we had hypothesized, answered incorrectly, we decided to test out another of our hypotheses.

One of our hypotheses was that, maybe, the reason our participants answered incorrectly was due to their young age. Their cognitive sophistication and their ability to logically reason were not as ripe as adults, making them answer incorrectly. Thus, we asked the same 2nd question to 2 adults and recorded their answers. Our adult participants, similar to our high school student ones, weren’t able to answer correctly.

As now we have our data, we can start to evaluate what might have caused these results.

Hypotheses Evaluation

We saw that our hypothesis as to how accurate our participants would be was not wrong and in fact, the disparity was a lot larger than what we had initially predicted. Now, we will go over each of our hypotheses one by one. Utilizing the answers given to the 3rd question of the survey and going over our participant profiles, we will try to find the most plausible reasons as to why our participants failed to find the correct answer.

Our participants were simply bad at mathematics and probability calculations.

We believe that this is not the most likely case as we will elaborate shortly. 34 of the high school students that participated in this survey were studying in Üsküdar American Academy (UAA), Turkey. In Turkey, to get into a prestigious and academically capable high school -UAA is regarded as one-, students have to take an exam called the LGS (the translation of the Turkish abbreviation being: Entrance Exam for High Schools). The LGS has a challenging mathematics section which has a country national average of around 3.31 correct answers out of 20 questions. The mathematics section consists of numerous topics such as algebra, probability, logical reasoning, geometry, and evaluation of rational expressions. To get into UAA, students either have to correctly answer all or nearly all of the 20 questions in the mathematics section. Although 3 of our participants were not attending this school, we do not believe that it had a large impact on the end result as 85% of students studying in UAA didn’t get the correct answer as well.

Therefore, we do not believe that our participants were inherently bad at mathematics and that the reason they failed to answer the question correctly was related to their mathematical capabilities or their ability to logically reason.

As the test had a 99% rate of accuracy, they thought how prevalent it is in the population was redundant in the probability evaluation.

Before evaluating whether or not we believe this hypothesis holds true, we would like to share some answers given to the 3rd question which asked participants why they answered the way they did.

Some answers they gave were:

• I chose 99% because I believe the 2nd statement about how much people have it was only given to confuse us.
• If my test was positive, then the amount of people who have the disease doesn’t concern me when doing the calculation.
• The number of people who have the disease does not affect the result.
• As the test is 99% accurate, it doesn’t matter how many people actually have the disease.
• No matter how many people have the disease, as our test result was positive we too are sick with 99% probability.

Observing the answers given to the question asking why they answered the way they did, we can see a clear thought pattern among our participants. Despite reading and seeing that the information regarding how prevalent the disease is was given, participants tended to disregard it. Participants were inclined to think that as the test already had an accuracy rate of 99%, the chances that you were sick wouldn’t be affected no matter how many people in the population were infected as well. Proving that a common fallacy among our participants was that they either thought the prevalence of the disease in the general population was redundant, or it was only a mere distraction to intentionally push them to give the wrong answer.

In the end, helping us say with confidence that our hypothesis proposing that people tended to disregard how many people were infected was, in fact, true.

Despite having mathematics and probability calculation knowledge, our participants weren’t aware of Bayes’ Theorem thus failed to choose the correct answer.

Although we never asked any of our high school student participants whether or not they knew or even heard what Bayes’ Theorem was, we believe it wouldn’t be that wrong of us to assume that they most probably weren’t as they were only high-school students. Taking this into consideration, we decided to test out whether or not a person’s prior knowledge about Bayes’ Theorem would make it more likely for them to answer correctly. Luckily, we had a participant who was one of our 2 adult participants who had learned about Bayes’ Theorem back in university. Before asking the question to our participant, we didn’t say anything about Bayes’ Theorem or anything else related to probability calculation. We only told our participant that they were being asked this question because it will be used for an experiment. Surprisingly, our participant answered wrong. After taking his answer, we asked them whether or not they had heard something called “Bayes’ Theorem”. They said they did a long time ago but never thought of it when asked the question.

As a result, we can postulate that even the knowledge of Bayes’ Theorem might not help out people as they might not draw the relationship between the question and the need to utilize the theorem.

Our participants were arguably young, thus, it might have caused their logical reasoning to not be as good as adults.

Thinking of this possibility, we asked 2 more participants to join the experiment, both adults. However, even though, as mentioned in the previous section, one of our adult participants knew Bayes’ Theorem, our adult participants as well didn’t answer correctly, making us conclude that even though there might still be a correlation between age and the ability to logically reason, it did not affect the outcome of the experiment.

Evaluating Participants Who Answered Correctly

Out of all 39 participants, only 4 answered correctly. As the 3rd question which asked participants to explain why they chose the answer they chose was optional, only 1 participant who answered correctly explained their answer.

The answer they provided was:

Actually the thing I thought was: sometimes probabilities which seem high to us are in actuality not that high. In reality, the answer I found was 9.9%. The probability that I am sick (%0.1) x the probability that the test is correct (%99). I multiplied them because there was a relation between them corresponding to one another

We can observe that the participant utilized somewhat of a primitive Bayes’ Theorem, postulating that they thought there was indeed a correlation between the probability of a person being sick and the probability of the test being correct. Which, in fact, is simply what Bayes’ Theorem is about! This answer might help us think a bit further as to the ability of people to be inherently good or bad at probability calculations, somewhat creating more questions than answers.

Ways of Improving Our Experiment & Conclusion

After viewing the data and making conclusions regarding whether or not our hypotheses were correct or not, there are a few more points to consider and address. These points are mainly regarding what might have faulted the data or what we might have done better when experimenting.

Some points that might have faulted the data are regarding the fact that our sample size might not have been that diverse or representative of a general populace. Our experiment utilized 39 participants in total, 37 of them only being high school students. Should we have been able to conduct this experiment in a larger and more representative populace/test group, although we still believe the results would have been the same, we could call our experiment to be a bit more representative and scientific.

Furthermore, we might have changed how we conducted our experiment and this might have resulted in a more natural environment for our participants. For example, in the 2nd question for the sake of simplicity, we decided to give our participants only 5 options to choose from. We might have not given them options to choose from and just asked for their unfiltered responses. We believe that it is a highly probable outcome that some of our participants might have been inclined to anchor their responses based on the options provided which might have disrupted the course of the experiment. Although there possibly were some points that could have been addressed in more detail and expertise, we believe that our experiment has yielded positive results. We were able to deduce that more times than none, the reason why people are unable to find the correct answer was that they failed to consider the number of people who were sick in the general population. Moreover, we saw some interesting results when it came to how people who previously knew about Bayes’ Theorem failed to utilize it.

In the end, we believe that although people are beings who can understand, process, and perform complex and intricate tasks, we all might have some problems or make mistakes when it comes to specific areas in life. One of these being the calculation of probability when 2 events related to each other are actually to consider instead of just one.

Even though our work was not evolutionary nor groundbreaking, we hope to have used the scientific method correctly to the utmost of our ability and understanding, hoping that through the better understanding of people and how we go about making decisions in life, we all can ensure that the society we live in is the most equitable and fair to the highest degree.

Works Cited

“Bayes Theorem.” YouTube, YouTube, 22 Dec. 2019, www.youtube.com/watch?v=HZGCoVF3YvM&t=605s.

1veritasium. “The Bayesian Trap.” YouTube, YouTube, 5 Apr. 2017, www.youtube.com/watch?v=R13BD8qKeTg&t=125s.

“LGS’de Derslerin Doğru Cevap Ortalaması Açıklandı.” Anasayfaya Dönmek Için Tıklayın, 30 June 2021, www.cumhuriyet.com.tr/haber/lgsde-derslerin-dogru-cevap-ortalamasi-aciklandi-1848565.