Android – Detect Open or Closed Eyes Using ML Kit and CameraX

Google’s ML kit is one of the best-trained models for face detection and its characteristics. Integrating with your own CameraX library can be quite a challenging task. so we are going to build an Android app that will detect whether a person’s eyes are open or closed in real time. This process going to be long so without delay let’s deep dive into the project. A sample video is given below to get an idea about what we are going to do in this article.

Note: Before starting the project please read about how the ML Kit detects faces and how cameraX works.

Project Setup

• Start a project with an empty activity and name your project whatever you want we are naming it EyeDetection.
• Language using Kotlin
• Minimum SDK set to Android 7.0(Nougat)

Step by Step Implementation

Adding Dependencies and Permissions

Open project-level build.gradle file, make sure to include Google’s Maven repository in both your buildscript and all projects sections. Add the dependencies for the ML Kit Android libraries to the module’s app-level gradle file, which is usually app/build.gradle.

dependencies {

  // This dependency will dynamically download the model in Google Play Services
  implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0'
  
  // camera dependencies 
  def camerax_version = "1.2.2"
  implementation "androidx.camera:camera-core:${camerax_version}"
  implementation "androidx.camera:camera-camera2:${camerax_version}"
  implementation "androidx.camera:camera-lifecycle:${camerax_version}"
  implementation "androidx.camera:camera-view:${camerax_version}"

}

Add this code in your manifest file

<uses-feature android:name="android.hardware.camera.any" />
<uses-permission android:name="android.permission.CAMERA" />

Add the following declaration to your app’s AndroidManifest.xml file. This will automatically download the model to the device if your app is installed from the Play Store.

<application ...>
      ...
      <meta-data
          android:name="com.google.mlkit.vision.DEPENDENCIES"
          android:value="face" >
      <!-- To use multiple models: android:value="face,model2,model3" -->
</application>

We are going to use viewbinding so don’t forget to enable it

buildFeatures {
   viewBinding true
}

Configure the Layout

Open the activity_main layout file at res/layout/activity_main.xml, and replace it with the following code.

XML

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout 
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <FrameLayout
        android:id="@+id/frameLayout"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent">

        <androidx.camera.view.PreviewView
            android:id="@+id/previewView_finder"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            app:scaleType="fillCenter">

        </androidx.camera.view.PreviewView>

        <com.example.eyedetection.GraphicOverlay
            android:id="@+id/graphicOverlay_finder"
            android:layout_width="match_parent"
            android:layout_height="match_parent" />

    </FrameLayout>

    <TextView
        android:id="@+id/tvWarningText"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="150dp"
        android:background="@color/white"
        android:focusableInTouchMode="false"
        android:gravity="center"
        android:padding="20dp"
        android:text="No Face detected"
        android:textColor="@android:color/holo_red_dark"
        android:textSize="18sp"
        android:visibility="visible"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent" />

</androidx.constraintlayout.widget.ConstraintLayout>

This code includes the PreviewView for preview of cameraX that will let the user to preview the photo they will be taking. A textview for the indication of face detection and about eyes, whether they are open or closed and GraphicOverlay to draw the box on detected faces.

Note: This GrapichOverlay view is a custom view, that you have to create a first.

Making Classes

Create a GraphicOverlay class and make It open. We have to define some methods and logics to draw over the screen. Below is the code.

Kotlin

import android.content.Context
import android.content.res.Configuration
import android.graphics.Canvas
import android.graphics.Rect
import android.graphics.RectF
import android.util.AttributeSet
import android.view.View
import kotlin.math.ceil

open class GraphicOverlay(context: Context?, attrs: AttributeSet?) :
    View(context, attrs) {
    private val lock = Any()
    private val faceBoxes: MutableList<FaceBox> = ArrayList()
    var mScale: Float? = null
    var mOffsetX: Float? = null
    var mOffsetY: Float? = null

    abstract class FaceBox(private val overlay: GraphicOverlay) {

        abstract fun draw(canvas: Canvas?)

        fun calculateRect(height: Float, width: Float, boundingBoxT: Rect): RectF {
            // for land scape
            fun isLandScapeMode(): Boolean {
                return overlay.context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE
            }

            fun whenLandScapeModeWidth(): Float {
                return when(isLandScapeMode()) {
                    true -> width
                    false -> height
                }
            }

            fun whenLandScapeModeHeight(): Float {
                return when(isLandScapeMode()) {
                    true -> height
                    false -> width
                }
            }

            val scaleX = overlay.width.toFloat() / whenLandScapeModeWidth()
            val scaleY = overlay.height.toFloat() / whenLandScapeModeHeight()
            val scale = scaleX.coerceAtLeast(scaleY)
            overlay.mScale = scale

            // Calculate offset (we need to center the overlay on the target)
            val offsetX = (overlay.width.toFloat() - ceil(whenLandScapeModeWidth() * scale)) / 2.0f
            val offsetY = (overlay.height.toFloat() - ceil(whenLandScapeModeHeight() * scale)) / 2.0f

            overlay.mOffsetX = offsetX
            overlay.mOffsetY = offsetY

            val mappedBox = RectF().apply {
                left = boundingBoxT.right * scale + offsetX
                top = boundingBoxT.top * scale + offsetY
                right = boundingBoxT.left * scale + offsetX
                bottom = boundingBoxT.bottom * scale + offsetY
            }
            return mappedBox
        }
    }

    fun clear() {
        synchronized(lock) { faceBoxes.clear() }
        postInvalidate()
    }

    fun add(faceBox: FaceBox) {
        synchronized(lock) { faceBoxes.add(faceBox) }
    }

    override fun onDraw(canvas: Canvas) {
        super.onDraw(canvas)
        synchronized(lock) {
            for (graphic in faceBoxes) {
                graphic.draw(canvas)
            }
        }
    }
}

To handle the camera we will create a cameraManager class to start the camera. Here we have not implemented the Picture taking ability as we are only doing real time detection so only preview will enough for us. CameraManager class takes five parameters context, previewView for the Preview, lifecycleOwner for the life cycle of camera. Listener to get the Status(which is an another class to track the status and change the textview). And graphicOverlay. For the analyzer we are using custom analyzer for face detection and draw boxes. Here is the code of cameraManager class.

Kotlin

import android.content.Context
import android.util.Log
import androidx.camera.core.Camera
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.ImageCapture
import androidx.camera.core.Preview
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.view.PreviewView
import androidx.core.content.ContextCompat
import androidx.lifecycle.LifecycleOwner
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors

class CameraManager(
    private val context: Context,
    private val previewView: PreviewView,
    private val lifecycleOwner: LifecycleOwner,
    private val graphicOverlay: GraphicOverlay,
    private val listener: ((Status) -> Unit),
) {

    private var preview: Preview? = null
    private var imageCapture: ImageCapture? = null
    private var camera: Camera? = null
    private lateinit var cameraExecutor: ExecutorService
    private var cameraSelectorOption = CameraSelector.LENS_FACING_BACK
    private var cameraProvider: ProcessCameraProvider? = null
    private var imageAnalyzer: ImageAnalysis? = null

    init {
        createNewExecutor()
    }

    private fun createNewExecutor() {
        cameraExecutor = Executors.newSingleThreadExecutor()
    }

    fun startCamera() {
        val cameraProviderFuture = ProcessCameraProvider.getInstance(context)
        cameraProviderFuture.addListener(
            {
                cameraProvider = cameraProviderFuture.get()
                preview = Preview.Builder().build()

                imageCapture = ImageCapture.Builder().build()

                imageAnalyzer = ImageAnalysis.Builder()
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build().also {
                        it.setAnalyzer(cameraExecutor, selectAnalyzer())
                    }

                val cameraSelector =
                    CameraSelector.Builder().requireLensFacing(cameraSelectorOption).build()

                setCameraConfig(cameraProvider, cameraSelector)

            }, ContextCompat.getMainExecutor(context)
        )
    }

    // Custom analyzer
    private fun selectAnalyzer(): ImageAnalysis.Analyzer {
        return FaceDetection(graphicOverlay, listener)
    }

    private fun setCameraConfig(
        cameraProvider: ProcessCameraProvider?, cameraSelector: CameraSelector
    ) {
        try {
            cameraProvider?.unbindAll()
            camera = cameraProvider?.bindToLifecycle(
                lifecycleOwner, cameraSelector, preview, imageCapture, imageAnalyzer
            )
            preview?.setSurfaceProvider(
                previewView.surfaceProvider
            )
        } catch (e: Exception) {
            Log.e("Error", "Use case binding failed", e)
        }
    }
}

This is the Status class to get the indication call back directly in our activity. that will be helpful to get the status and change the textview according to analyzer.

Kotlin

enum class Status { NO_FACE, MULTIPLE_FACES, LEFT_EYE_CLOSED,
    RIGHT_EYE_CLOSED, BOTH_EYES_CLOSED,VALID_FACE
}

This is the face detection class that will be used by our custom analyzer. Which is inheriting the Analyzer class. And overriding its methods. The code doesn’t draw the box if multiple faces detected but will notifiy the main activity using the listener. And if it is a a single face , we will pass its properties to FaceBox class to draw the box and for eye detection.

Kotlin

import android.graphics.Rect
import android.util.Log
import androidx.camera.core.ImageProxy
import com.google.android.gms.tasks.Task
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.face.Face
import com.google.mlkit.vision.face.FaceDetection
import com.google.mlkit.vision.face.FaceDetectorOptions
import java.io.IOException

class FaceDetection(
    private val graphicOverlayView: GraphicOverlay,
    private val listener: (Status) -> Unit
) : Analyzer<List<Face>>() {

    private val realTimeOpts =
        FaceDetectorOptions.Builder().setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
            .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
            .setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL).build()

    private val detector = FaceDetection.getClient(realTimeOpts)

    override val graphicOverlay: GraphicOverlay
        get() = graphicOverlayView

    override fun detectInImage(image: InputImage): Task<List<Face>> {
        return detector.process(image)
    }

    override fun stop() {
        try {
            detector.close()
        } catch (e: IOException) {
            Log.e("Error", "Exception thrown while trying to close Face Detector: $e")
        }
    }

    override fun onSuccess(
        results: List<Face>, graphicOverlay: GraphicOverlay, rect: Rect, imageProxy: ImageProxy
    ) {
        graphicOverlay.clear()
        // If multiple faces then don't draw
        if (results.isNotEmpty()) {
            if (results.size > 1) {
                listener(Status.MULTIPLE_FACES)
            } else {
                for (face in results) {
                    val faceGraphic =
                        FaceBox(graphicOverlay, face, rect, listener)
                    graphicOverlay.add(faceGraphic)
                }
            }
            graphicOverlay.postInvalidate()
        } else {
            listener(Status.NO_FACE)
            Log.e("Error", "Face Detector failed.")
        }
    }

    override fun onFailure(e: Exception) {
        Log.e("Error", "Face Detector failed. $e")
        listener(Status.NO_FACE)
    }
}

The faceBox class inherting the Graphic overlay class. This class will calculate the probability of eyes of this face and also put the green color box to the detected face. We have set the probability to 0.6. if any eye probability is less than or equal to 0.6 we will consider that eye as closed eye and set status accordingly. Here is code of FaceBox class.

Kotlin

import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import android.graphics.Rect
import com.google.mlkit.vision.face.Face

class FaceBox(
    overlay: GraphicOverlay,
    private val face: Face,
    private val imageRect: Rect,
    private val listener: (Status) -> Unit
) : GraphicOverlay.FaceBox(overlay) {

    private val facePositionPaint: Paint
    private val idPaint: Paint
    private val boxPaint: Paint

    init {
        val selectedColor = Color.WHITE
        facePositionPaint = Paint()
        facePositionPaint.color = selectedColor
        idPaint = Paint()
        idPaint.color = selectedColor
        boxPaint = Paint()
        boxPaint.color = selectedColor
        boxPaint.style = Paint.Style.STROKE
        boxPaint.strokeWidth = 5.0f
    }

    private val greenBoxPaint = Paint().apply {
        color = Color.GREEN
        style = Paint.Style.STROKE
        strokeWidth = 5.0f
    }

    override fun draw(canvas: Canvas?) {
        val rect = calculateRect(
            imageRect.height().toFloat(), imageRect.width().toFloat(), face.boundingBox
        )
        val leftEyeProbability = leftEyeProbability()
        val rightEyeProbability = rightEyeProbability()

        when {
            // both eyes are closed
            leftEyeProbability <= 0.6 && rightEyeProbability() <= 0.6 -> {
                listener(Status.BOTH_EYES_CLOSED)
            }
            // left eye is closed
            leftEyeProbability <= 0.6 -> {
                listener(Status.LEFT_EYE_CLOSED)
            }
            // right is closed
            rightEyeProbability <= 0.6 -> {
                listener(Status.RIGHT_EYE_CLOSED)
            }
            // valid face, set face box color green
            else -> {
                listener(Status.VALID_FACE)
                canvas?.drawRect(rect, greenBoxPaint)
            }
        }
    }
    private fun leftEyeProbability(): Float {
        var probability = 0.0F
        if (face.leftEyeOpenProbability != null) {
            val leftEyeOpenProb = face.leftEyeOpenProbability
            probability = leftEyeOpenProb!!
        }
        return probability
    }

    private fun rightEyeProbability(): Float {
        var probability = 0.0F
        if (face.rightEyeOpenProbability != null) {
            val rightEyeOpenProb = face.rightEyeOpenProbability
            probability = rightEyeOpenProb!!
        }
        return probability
    }
}

We are almost ready. Before starting the app we have to add the camera permission and handle it accordingly. And after handling the permission in our activity out code will look like this.

Kotlin

import android.Manifest
import android.content.ActivityNotFoundException
import android.content.Intent
import android.content.pm.PackageManager
import android.net.Uri
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.provider.Settings
import android.util.Log
import androidx.activity.result.contract.ActivityResultContracts
import androidx.appcompat.app.AlertDialog
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import com.example.eyedetection.databinding.ActivityMainBinding

class MainActivity : AppCompatActivity() {
    private lateinit var binding: ActivityMainBinding
    private lateinit var cameraManager: CameraManager

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(binding.root)
        createCameraManager()
        // to handle the permission
        cameraPermission()
    }

    private fun openCamera() {
        // this will start the camera if permission is enabled
        cameraManager.startCamera()
    }

    private fun cameraPermission() {
        val cameraPermission = Manifest.permission.CAMERA

        if (ContextCompat.checkSelfPermission(
                this, cameraPermission
            ) == PackageManager.PERMISSION_GRANTED
        ) {
            openCamera()
        } else if (ActivityCompat.shouldShowRequestPermissionRationale(
               this, cameraPermission
            )
        ) {
            val title = "Permission Required"
            val message = "App needs Camera Permission to detect faces"
            val builder: AlertDialog.Builder = AlertDialog.Builder(this)
            builder.setTitle(title).setMessage(message).setCancelable(false)
                .setPositiveButton("OK") { dialog, _ ->
                    requestCameraPermissionLauncher.launch(cameraPermission)
                    dialog.dismiss()
                }.setNegativeButton("Cancel") { dialog, _ ->
                    dialog.dismiss()
                }
            builder.create().show()
        } else {
            requestCameraPermissionLauncher.launch(
                cameraPermission
            )
        }
    }

    private val requestCameraPermissionLauncher = registerForActivityResult(
        ActivityResultContracts.RequestPermission()
    ) { isGranted: Boolean ->
        if (isGranted) {
            openCamera()
        } else if (!ActivityCompat.shouldShowRequestPermissionRationale(
               this, Manifest.permission.CAMERA
            )
        ) {
            val title = "Permission required"
            val message =
                "Please allow camera permission to detect faces"
            val builder: AlertDialog.Builder = AlertDialog.Builder(this)
            builder.setTitle(title).setMessage(message).setCancelable(false)
                .setPositiveButton("Change Settings") { _, _ ->
                    try {
                        val intent = Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS)
                        val uri = Uri.fromParts("package", this.packageName, null)
                        intent.data = uri
                        startActivity(intent)
                    } catch (e: ActivityNotFoundException) {
                        e.printStackTrace()
                    }
                }.setNegativeButton("Cancel") { dialog, _ ->
                    dialog.dismiss()
                }
            builder.create().show()
        } else {
            cameraPermission()
        }
    }
    private fun createCameraManager() {
        cameraManager = CameraManager(
            this,
            binding.previewViewFinder,
            this,
            binding.graphicOverlayFinder,
            ::checkStatus
        )
    }

    private fun checkStatus(status: Status) {
        Log.e("status","$status")
        when (status) {
            Status.MULTIPLE_FACES -> {
                binding.tvWarningText.text = "Multiple Faces detected"
            }
            Status.NO_FACE -> {
                binding.tvWarningText.text = "No Face detected"
            }
            Status.LEFT_EYE_CLOSED -> {
                binding.tvWarningText.text = "Left eye is closed"
            }
            Status.RIGHT_EYE_CLOSED -> {
                binding.tvWarningText.text ="Right eye is closed"
            }
            Status.BOTH_EYES_CLOSED->{
                binding.tvWarningText.text = "Both Eyes are closed"
            }
            Status.VALID_FACE ->{
                binding.tvWarningText.text ="Correct Face"
            }
        }
    }
}

checkStatus function will listen to the listener and set the text of textview as soon as any face or any changes in face is detected. And now we are finally ready. Build the app and run in your device. Our final result will be look like this.

Output: