Open In App

Logistic Regression using Statsmodels

Prerequisite: Understanding Logistic Regression
Logistic regression is the type of regression analysis used to find the probability of a certain event occurring. It is the best suited type of regression for cases where we have a categorical dependent variable which can take only discrete values. 

The dataset : 
In this article, we will predict whether a student will be admitted to a particular college, based on their gmat, gpa scores and work experience. The dependent variable here is a Binary Logistic variable, which is expected to take strictly one of two forms i.e., admitted or not admitted

Building the Logistic Regression model :

Statsmodels is a Python module that provides various functions for estimating different statistical models and performing statistical tests  




# importing libraries
import statsmodels.api as sm
import pandas as pd 
  
# loading the training dataset 
df = pd.read_csv('logit_train1.csv', index_col = 0)
  
# defining the dependent and independent variables
Xtrain = df[['gmat', 'gpa', 'work_experience']]
ytrain = df[['admitted']]
   
# building the model and fitting the data
log_reg = sm.Logit(ytrain, Xtrain).fit()

Output : 

Optimization terminated successfully.
         Current function value: 0.352707
         Iterations 8

In the output, ‘Iterations‘ refer to the number of times the model iterates over the data, trying to optimize the model. By default, the maximum number of iterations performed is 35, after which the optimization fails.

The summary table :

The summary table below gives us a descriptive summary about the regression results.  




# printing the summary table
print(log_reg.summary())

Output : 

                           Logit Regression Results                           
==============================================================================
Dep. Variable:               admitted   No. Observations:                   30
Model:                          Logit   Df Residuals:                       27
Method:                           MLE   Df Model:                            2
Date:                Wed, 15 Jul 2020   Pseudo R-squ.:                  0.4912
Time:                        16:09:17   Log-Likelihood:                -10.581
converged:                       True   LL-Null:                       -20.794
Covariance Type:            nonrobust   LLR p-value:                 3.668e-05
===================================================================================
                      coef    std err          z      P>|z|      [0.025      0.975]
-----------------------------------------------------------------------------------
gmat               -0.0262      0.011     -2.383      0.017      -0.048      -0.005
gpa                 3.9422      1.964      2.007      0.045       0.092       7.792
work_experience     1.1983      0.482      2.487      0.013       0.254       2.143
===================================================================================

Explanation of some of the terms in the summary table:

Predicting on New Data :

Now we shall test our model on new test data. The test data is loaded from this csv file.
The predict() function is useful for performing predictions. The predictions obtained are fractional values(between 0 and 1) which denote the probability of getting admitted. These values are hence rounded, to obtain the discrete values of 1 or 0. 




# loading the testing dataset  
df = pd.read_csv('logit_test1.csv', index_col = 0)
  
# defining the dependent and independent variables
Xtest = df[['gmat', 'gpa', 'work_experience']]
ytest = df['admitted']
  
# performing predictions on the test dataset
yhat = log_reg.predict(Xtest)
prediction = list(map(round, yhat))
  
# comparing original and predicted values of y
print('Actual values', list(ytest.values))
print('Predictions :', prediction)

Output : 

Optimization terminated successfully.
         Current function value: 0.352707
         Iterations 8
Actual values [0, 0, 0, 0, 0, 1, 1, 0, 1, 1]
Predictions : [0, 0, 0, 0, 0, 0, 0, 0, 1, 1]

Testing the accuracy of the model : 




from sklearn.metrics import (confusion_matrix, 
                           accuracy_score)
  
# confusion matrix
cm = confusion_matrix(ytest, prediction) 
print ("Confusion Matrix : \n", cm) 
  
# accuracy score of the model
print('Test accuracy = ', accuracy_score(ytest, prediction))

Output : 

Confusion Matrix : 
 [[6 0]
 [2 2]]
Test accuracy =  0.8

 


Article Tags :