Finding the probability of a state at a given time in a Markov chain | Set 2

Given a Markov chain G, we have the find the probability of reaching the state F at time t = T if we start from state S at time t = 0.

A Markov chain is a random process consisting of various states and the probabilities of moving from one state to another. We can represent it using a directed graph where the nodes represent the states and the edges represent the probability of going from one node to another. It takes unit time to move from one node to another. The sum of the associated probabilities of the outgoing edges is one for every node.

Consider the given Markov Chain( G ) as shown in below image:

Examples:

Input : S = 1, F = 2, T = 1
Output: 0.23
We start at state 1 at t = 0, 
so there is a probability of 0.23 
that we reach state 2 at t = 1.

Input: S = 4, F = 2, T = 100
Output: 0.284992

In the previous article, a dynamic programming approach is discussed with a time complexity of O(N2T), where N is the number of states.



Matrix exponentiation approach: We can make an adjacency matrix for the Markov chain to represent the probabilities of transitions between the states. For example, the adjacency matrix for the graph given above is:

     \[    M=   \left[ {\begin{array}{cccccc}    0 & 0.09 & 0 & 0 & 0 & 0 \\    0.23 & 0 & 0 & 0 & 0 & 0.62 \\    0 & 0.06 & 0 & 0 & 0 & 0 \\    0.77 & 0 & 0.63 & 0 & 0 & 0 \\    0 & 0 & 0 & 0.65 & 0 & 0.38 \\    0 & 0.85 & 0.37 & 0.35 & 1.0 & 0 \\   \end{array} } \right] \]

We can observe that the probability distribution at time t is given by P(t) = M * P(t – 1), and the initial probability distribution P(0) is a zero vector with the Sth element being one. Using these results, we can get solve the recursive expression for P(t). For example, if we take S to be 3, then P(t) is given by,

     \[    P(t)= M^t   \left[ {\begin{array}{c}    0 \\    0 \\    1 \\    0 \\    0 \\    0 \\   \end{array} } \right] \]

If we use effective matrix exponentiation technique, then the time complexity of this approach comes out to be O(N3 * log T). This approach performs better than the dynamic programming approach if the value of T is considerably higher than the number of states, i.e. N.

Below is the implementation of the above approach:

C++

filter_none

edit
close

play_arrow

link
brightness_4
code

// C++ implementation of the above approach
#include <bits/stdc++.h>
using namespace std;
  
// Macro to define a vector of float
#define vf vector<float>
  
// Function to multiply two matrices A and B
vector<vf > multiply(vector<vf > A,    vector<vf > B, int N)
{
    vector<vf > C(N, vf(N, 0));
    for (int i = 0; i < N; ++i)
        for (int j = 0; j < N; ++j)
            for (int k = 0; k < N; ++k)
                C[i][j] += A[i][k] * B[k][j];
    return C;
}
  
// Function to calculate the power of a matrix
vector<vf > matrix_power(vector<vf > M, int p, int n)
{
    vector<vf > A(n, vf(n, 0));
    for (int i = 0; i < n; ++i)
        A[i][i] = 1;
  
    while (p) {
        if (p % 2)
            A = multiply(A, M, n);
        M = multiply(M, M, n);
        p /= 2;
    }
  
    return A;
}
  
// Function to calculate the probability of 
// reaching F at time T after starting from S
float findProbability(vector<vf > M, int N, int F, 
                                        int S, int T)
{
    // Storing M^T in MT
    vector<vf > MT = matrix_power(M, T, N);
  
    // Returning the answer
    return MT[F - 1][S - 1];
}
  
// Driver code
int main()
{
    // Adjacency matrix
    // The edges have been stored in the row
    // corresponding to their end-point
    vector<vf > G{ { 0, 0.09, 0, 0, 0, 0 },
                   { 0.23, 0, 0, 0, 0, 0.62 },
                   { 0, 0.06, 0, 0, 0, 0 },
                   { 0.77, 0, 0.63, 0, 0, 0 },
                   { 0, 0, 0, 0.65, 0, 0.38 },
                   { 0, 0.85, 0.37, 0.35, 1.0, 0 }};
  
    // N is the number of states
    int N = 6;
      
    int S = 4, F = 2, T = 100;
  
    cout << "The probability of reaching " << F << " at time " 
                << T << "\nafter starting from " << S << " is " 
                << findProbability(G, N, F, S, T);
  
    return 0;
}

chevron_right


Java

filter_none

edit
close

play_arrow

link
brightness_4
code

// Java implementation of the above approach 
class GFG 
{
  
    // Function to multiply two matrices A and B
    static double[][] multiply(double[][] A,
            double[][] B, int N) 
    {
        double[][] C = new double[N][N];
        for (int i = 0; i < N; ++i)
            for (int j = 0; j < N; ++j)
                for (int k = 0; k < N; ++k)
                    C[i][j] += A[i][k] * B[k][j];
        return C;
    }
  
    // Function to calculate the power of a matrix
    static double[][] matrix_power(double[][] M, int p, int n) 
    {
        double[][] A = new double[n][n];
        for (int i = 0; i < n; ++i)
            A[i][i] = 1;
  
        while (p > 0
        {
            if (p % 2 == 1)
                A = multiply(A, M, n);
            M = multiply(M, M, n);
            p /= 2;
        }
        return A;
    }
  
    // Function to calculate the probability of
    // reaching F at time T after starting from S
    static double findProbability(double[][] M, 
                    int N, int F, int S, int T)
    {
        // Storing M^T in MT
        double[][] MT = matrix_power(M, T, N);
  
        // Returning the answer
        return MT[F - 1][S - 1];
    }
  
    // Driver code
    public static void main(String[] args) 
    {
        // Adjacency matrix
        // The edges have been stored in the row
        // corresponding to their end-point
        double[][] G = { { 0, 0.09, 0, 0, 0, 0 },
                        { 0.23, 0, 0, 0, 0, 0.62 },
                        { 0, 0.06, 0, 0, 0, 0 },
                        { 0.77, 0, 0.63, 0, 0, 0 },
                        { 0, 0, 0, 0.65, 0, 0.38 },
                        { 0, 0.85, 0.37, 0.35, 1.0, 0 } };
  
        // N is the number of states
        int N = 6;
  
        int S = 4, F = 2, T = 100;
  
        System.out.printf(
                "The probability of reaching " + F + 
                " at time " + T + "\nafter starting from "
                        S + " is %f",
                findProbability(G, N, F, S, T));
    }
}
  
// This code is contributed by Rajput-Ji

chevron_right


C#

filter_none

edit
close

play_arrow

link
brightness_4
code

// C# implementation of the above approach 
using System;
  
class GFG 
{
  
    // Function to multiply two matrices A and B
    static double[,] multiply(double[,] A,
                            double[,] B, int N) 
    {
        double[,] C = new double[N, N];
        for (int i = 0; i < N; ++i)
            for (int j = 0; j < N; ++j)
                for (int k = 0; k < N; ++k)
                    C[i, j] += A[i, k] * B[k, j];
        return C;
    }
  
    // Function to calculate the power of a matrix
    static double[,] matrix_power(double[,] M, int p, int n) 
    {
        double[,] A = new double[n,n];
        for (int i = 0; i < n; ++i)
            A[i, i] = 1;
  
        while (p > 0) 
        {
            if (p % 2 == 1)
                A = multiply(A, M, n);
            M = multiply(M, M, n);
            p /= 2;
        }
        return A;
    }
  
    // Function to calculate the probability of
    // reaching F at time T after starting from S
    static double findProbability(double[,] M, 
                    int N, int F, int S, int T)
    {
        // Storing M^T in MT
        double[,] MT = matrix_power(M, T, N);
  
        // Returning the answer
        return MT[F - 1, S - 1];
    }
  
    // Driver code
    public static void Main(String[] args) 
    {
        // Adjacency matrix
        // The edges have been stored in the row
        // corresponding to their end-point
        double[,] G = { { 0, 0.09, 0, 0, 0, 0 },
                        { 0.23, 0, 0, 0, 0, 0.62 },
                        { 0, 0.06, 0, 0, 0, 0 },
                        { 0.77, 0, 0.63, 0, 0, 0 },
                        { 0, 0, 0, 0.65, 0, 0.38 },
                        { 0, 0.85, 0.37, 0.35, 1.0, 0 } };
  
        // N is the number of states
        int N = 6;
  
        int S = 4, F = 2, T = 100;
  
        Console.Write("The probability of reaching " + F + 
                " at time " + T + "\nafter starting from "
                        S + " is {0:F6}",
                findProbability(G, N, F, S, T));
    }
}
  
// This code is contributed by 29AjayKumar

chevron_right


Output:

The probability of reaching 2 at time 100
after starting from 4 is 0.284991

Time Complexity: O(N3 * logT)
Space Complexity: O(N2)

Attention reader! Don’t stop learning now. Get hold of all the important DSA concepts with the DSA Self Paced Course at a student-friendly price and become industry ready.




My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.