Bayes’ Theorem

Bayes’ Theorem is named after Reverend Thomas Bayes. It is a very important theorem in mathematics that is used to find the probability of an event, based on prior knowledge of conditions that might be related to that event. It is a further case of conditional probability. For example, There are 3 bags, each containing some white marble and some black marble in each bag. If a white marble is drawn at random. What is the probability to find that this white marble is from the first bag? In cases like such, we use Bayes’ Theorem. It is used where the probability of occurrence of a particular event is calculated based on other conditions which are also called conditional probability.

What is Bayes’ Theorem?

Bayes theorem is also known as the Bayes Rule or Bayes Law. It is used to determine the conditional probability of event A when event B has already happened. The general statement of Bayes’ theorem is “The conditional probability of an event A, given the occurrence of another event B, is equal to the product of the event of B, given A and the probability of A divided by the probability of event B.” i.e.

P(A|B) = P(B|A)P(A) / P(B)

where,
P(A) and P(B) are the probabilities of events A and B
P(A|B) is the probability of event A when event B happens
P(B|A) is the probability of event B when A happens

Bayes Theorem Statement

Bayes’ Theorem for n set of events is defined as,

Let E₁, E₂,…, E_n be a set of events associated with the sample space S, in which all the events E₁, E₂,…, E_n have a non-zero probability of occurrence. All the events E₁, E₂,…, E form a partition of S. Let A be an event from space S for which we have to find probability, then according to Bayes’ theorem,

P(E_i|A) = P(E_i)P(A|E_i) / ∑ P(E_k)P(A|E_k)

for k = 1, 2, 3, …., n

Terms Related to Bayes Theorem

As we have studied about Bayes theorem in detail, let us understand the meanings of a few terms related to the concept which have been used in the Bayes theorem formula and derivation:

Conditional Probability

The probability of an event A based on the occurrence of another event B is termed conditional Probability. It is denoted as P(A|B) and represents the probability of A when event B has already happened.

Joint Probability

When the probability of two more events occurring together and at the same time is measured it is marked as Joint Probability. For two events A and B, it is denoted by joint probability is denoted as, P(A∩B).

Random Variables

Real-valued variables whose possible values are determined by random experiments are called random variables. The probability of finding such variables is the experimental probability.

Bayes Theorem Formula

For any two events A and B, then the formula for the Bayes theorem is given by: (the image given below gives the Bayes’ theorem formula)

where,
P(A) and P(B) are the probabilities of events A and B also P(B) is never equal to zero.
P(A|B) is the probability of event A when event B happens
P(B|A) is the probability of event B when A happens

Bayes Theorem Derivation

The proof of Bayes’ Theorem is given as, according to the conditional probability formula,

P(E_i|A) = P(E_i∩A) / P(A)…..(i)

Then, by using the multiplication rule of probability, we get

P(E_i∩A) = P(E_i)P(A|E_i)……(ii)

Now, by the total probability theorem,

P(A) = ∑ P(E_k)P(A|E_k)…..(iii)

Substituting the value of P(E_i∩A) and P(A) from eq (ii) and eq(iii) in eq(i) we get,

P(E_i|A) = P(E_i)P(A|E_i) / ∑ P(E_k)P(A|E_k)

Note:

Various terms used in  Bayes theorem are explained below in this article,

• Hypotheses: Events happening in the sample space E₁, E₂,… E_n is called the hypotheses
• Priori Probability: P(E_i) is known as the priori probability of hypothesis E_i.
• Posteriori Probability: Probability P(E_i|A) is considered as the posterior probability of hypothesis E_i

Bayes’ theorem is also known as the formula for the probability of “causes”. As we know that, the E_i‘s are a partition of the sample space S, and at any given time only one of the events E_i occurs. Thus we conclude that the Bayes’ theorem formula gives the probability of a particular E_i, given the event A has occurred.

Difference Between Conditional Probability and Bayes Theorem

The difference between Conditional Probability and Bayes Theorem can be understood with the help of the table given below,

Bayes’ Theorem	Conditional Probability
Bayes’ Theorem is derived using the definition of conditional probability. It is used to find the reverse probability.	Conditional Probability is the probability of event A when event B has already occurred.
Formula: P(A|B) = [P(B|A)P(A)] / P(B)	Formula: P(A|B) = P(A∩B) / P(B)

Examples of Bayes’ Theorem

Bayesian inference is very important and has found application in various activities, including medicine, science, philosophy, engineering, sports, law, etc. and Bayesian inference is directly derived from Bayes’ theorem. Example: Bayes’ theorem defines the accuracy of the medical test by taking into account how likely a person is to have a disease and what is the overall accuracy of the test.

Theorem of Total Probability

Let E₁, E₂,…………..E_n is mutually exclusive and exhaustive events associated with a random experiment and lets E be an event that occurs with some E_i. Then, prove that 

P(E) = ⁿ∑_i=1P(E/E_i) . P(E_j)

Proof:

Let S be the sample space. Then,

S = E₁ ∪ E₂ ∪ E₃  ∪………………… ∪ En and E_i ∩ E_j = ∅ for i ≠ j.

E = E∩S 
   = E ∩ (E₁ ∪ E₂ ∪ E₃  ∪………………… ∪ E_n)
   = (E ∩ E₁) ∪ (E ∩ E₂) ∪ ……∪ (E ∩ E_n)

P(E) = P{(E ∩ E₁) ∪ (E ∩ E₂)∪……∪(E ∩ E_n)}
       = P(E ∩ E₁) + P(E ∩ E₂) + …… + P(E ∩ E_n)
       = {Therefore, (E ∩ E₁), (E ∩ E₂),………….,(E ∩ E_n)} are pairwise disjoint}
       = P(E/E₁) . P(E₁) + P(E/E₂) . P(E₂) +……………………+ P(E/E_n) . P(E_n)  [by multiplication theorem]
       = ⁿ∑_i=1P(E/E_i) . P(E_i)

Read, More

• Permutations and Combinations
• Binomial Theorem
• Probability Distribution

Numerical Example of Bayes’ Theorem

Example 1: A person has undertaken a job. The probabilities of completion of the job on time with and without rain are 0.44 and 0.95 respectively. If the probability that it will rain is 0.45, then determine the probability that the job will be completed on time.

Solution:

Let E₁ be the event that the mining job will be completed on time and E₂ be the event that it rains. We have,

P(A) = 0.45,

P(no rain) = P(B) = 1 − P(A) = 1 − 0.45 = 0.55

By multiplication law of probability,

P(E₁) = 0.44

P(E₂) = 0.95

Since, events A and B form partitions of the sample space S, by total probability theorem, we have

P(E) = P(A) P(E₁) + P(B) P(E₂)

= 0.45 × 0.44 + 0.55 × 0.95

= 0.198 + 0.5225 = 0.7205

So, the probability that the job will be completed on time is 0.684.

Example 2: There are three urns containing 3 white and 2 black balls; 2 white and 3 black balls; 1 black and 4 white balls respectively. There is an equal probability of each urn being chosen. One ball is equal probability chosen at random. what is the probability that a  white ball is drawn?

Solution:

Let E₁, E₂, and E₃ be the events of choosing the first, second, and third urn respectively. Then,

P(E₁) = P(E₂) = P(E₃) =1/3

Let E be the event that a white ball is drawn. Then,

P(E/E₁) = 3/5, P(E/E₂) = 2/5, P(E/E₃) = 4/5

By theorem of total probability, we have

P(E) = P(E/E₁) . P(E₁) + P(E/E₂) . P(E₂) + P(E/E₃) . P(E₃)

       = (3/5 × 1/3) + (2/5 × 1/3) + (4/5 × 1/3)

       = 9/15 = 3/5

Example 3: A card from a pack of 52 cards is lost. From the remaining cards of the pack, two cards are drawn and are found to be both hearts. find the probability of the lost card being a heart.

Solution:

Let E₁, E₂, E_3, and E₄ be the events of losing a card of hearts, clubs, spades, and diamonds respectively.

Then P(E₁) = P(E₂) = P(E₃) = P(E₄) = 13/52 = 1/4.

Let E be the event of drawing 2 hearts from the remaining 51 cards. Then,

P(E|E₁) = probability of drawing 2 hearts, given that a card of hearts is missing

           = ¹²C₂ / ⁵¹C₂ = (12 × 11)/2! × 2!/(51 × 50) = 22/425

P(E|E₂) = probability of drawing 2 clubs ,given that a card of clubs is missing

           = ¹³C₂ / ⁵¹C₂ = (13 × 12)/2! × 2!/(51 × 50) = 26/425

P(E|E₃) = probability of drawing 2 spades ,given that a card of hearts is missing

           = ¹³C₂ / ⁵¹C₂ = 26/425

P(E|E₄) = probability of drawing 2 diamonds ,given that a card of diamonds is missing

           = ¹³C₂ / ⁵¹C₂ = 26/425

Therefore,

P(E₁|E) = probability of the lost card is being a heart, given the 2 hearts are drawn from the remaining 51 cards

           = P(E₁) . P(E|E₁)/P(E₁) . P(E|E₁) + P(E₂) . P(E|E₂) + P(E₃) . P(E|E₃) + P(E₄) . P(E|E₄)

           = (1/4 × 22/425) / {(1/4 × 22/425) + (1/4 × 26/425) + (1/4 × 26/425) + (1/4 × 26/425)}

           = 22/100 = 0.22

Hence, The required probability is 0.22.

Example 4: Suppose 15 men out of 300 men and 25 women out of 1000 are good orators. An orator is chosen at random. Find the probability that a male person is selected. Assume that there are equal numbers of men and women.

Solution:

Let there be 1000 men and 1000 women.

Let E₁ and E₂ be the events of choosing a man and a woman respectively. Then,

P(E₁) = 1000/2000 = 1/2 , and P(E₂) = 1000/2000 = 1/2

Let E be the event of choosing an orator. Then,

P(E|E₁) = 50/1000 = 1/20, and P(E|E₂) = 25/1000 = 1/40

Probability of selecting a male person ,given that the person selected is a good orator 

P(E₁/E) = P(E|E₁) × P(E₁)/ P(E|E₁) × P(E₁) + P(E|E₂) × P(E₂)

            = (1/2 × 1/20) /{(1/2 × 1/20) + (1/2 × 1/40)}

            = 2/3

Hence the required probability is 2/3.

Example 5: A man is known to speak the lies 1 out of 4 times. He throws a die and reports that it is a six. Find the probability that is actually a six.

Solution: 

In a throw of a die, let

E₁ = event of getting a six,

E₂ = event of not getting a six and

E = event that the man reports that it is a six.

Then, P(E₁) = 1/6, and P(E₂) = (1 – 1/6) = 5/6

P(E|E₁) = probability that the man reports that six occurs when six has actually occurred 

           = probability that the man speaks the truth

           = 3/4

P(E|E₂) = probability that the man reports that six occurs when six has not actually occurred

           = probability that the man does not speak the truth

           = (1 – 3/4) = 1/4

Probability of getting a six ,given that the man reports it to be six 

P(E₁|E) = P(E|E₁) × P(E₁)/P(E|E₁) × P(E₁) + P(E|E₂) × P(E₂)     [by Bayes’ theorem]

           = (3/4 × 1/6)/{(3/4 × 1/6) + (1/4 × 5/6)}

           = (1/8 × 3) = 3/8

Hence the probability required is 3/8.

FAQs on Bayes’ Theorem

Question 1: What is Bayes’ theorem?

Answer:

Bayes, theorem as the name suggest is a mathematical theorem which is used to find the conditionality probability of an event. Conditional probability is the probability of the event which will occur in future. It is calculated based on the previous outcomes of the events.

Question 2: When to use Bayes’ theorem?

Answer:

Bayes’ theorem is applicable when the conditional probability of an event is given, it is used to find the reverse probability of the event.

Question 3: How is Bayes’ theorem different from conditional probability?

Answer:

Bayes’ theorem is used to define the probability of an event based on the previous conditions of the event. Whereas, Bayes’ Theorem uses conditional probability to find the reverse probability of the event.

Question 4: What is the formula for Bayes’ theorem?

Answer:

Bayes theorem formula is explained below,

P(A|B) = [P(B|A) P(A)] / P(B)

where,
P(A) and P(B) are the probabilities of events A and B
P(A|B) is the probability of event A when event B happens
P(B|A) is the probability of event B when A happens