How to Label Image in Android using Firebase ML Kit?

We have seen many apps in Android in which we will detect the object present in the image whether it may be any object. In this article, we will take a look at the implementation of image labeling in Android using Firebase ML Kit. 

What we are going to build in this article? 

We will be building a simple application in which we will be capturing an image of any object and from that, we will detect the objects present inside our image with the accuracy level. A sample video is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using the Java language. 

Step by Step Implementation

Step 1: Create a New Project

To create a new project in Android Studio please refer to How to Create/Start a New Project in Android Studio. Note that select Java as the programming language.

Step 2: Connect your app to Firebase

After creating a new project in Android Studio connect your app to Firebase. For connecting your app to firebase. Navigate to Tools on the top bar. After that click on Firebase. A new window will open on the right side. Inside that window click on Firebase ML and then click on Use Firebase ML kit in Android. You can see the option below screenshot.  

After clicking on this option on the next screen click on Connect to Firebase option to connect your app to Firebase. 

Step 3: Adding dependency for language translation to build.gradle file

Navigate to the Gradle Scripts > build.gradle(Module:app) and add the below dependency in the dependencies section.   

// dependency for firebase core.

implementation’com.google.firebase:firebase-core:15.0.2′

// Firebase ML dependency

implementation ‘com.google.firebase:firebase-ml-vision:24.0.3’

implementation ‘com.google.firebase:firebase-ml-vision-image-label-model:20.0.1’

Step 4: Adding permissions to access the Internet and meta-data in your Android Apps AndroidManifest file  

Navigate to the app > AndroidManifest.xml file and add the below code to it. Comments are added in the code to get to know in more detail.  

XML

<!-- below line is use to add camera feature in our app -->
<uses-feature
  android:name="android.hardware.camera"
  android:required="true" />

<!-- permission for internet -->
<uses-permission android:name="android.permission.INTERNET" />

Add the below line inside your application tag. 

XML

<meta-data
  android:name="com.google.firebase.ml.vision.DEPENDENCIES"
  android:value="label" />

Below is the complete code for the AndroidManifest.xml file.

XML

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.labelimage">

    <!-- below line is use to add camera feature in our app -->
    <uses-feature
        android:name="android.hardware.camera"
        android:required="true" />

    <!-- permission for internet -->
    <uses-permission android:name="android.permission.INTERNET" />

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:supportsRtl="true"
        android:theme="@style/Theme.LabelImage">
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>

        <meta-data
            android:name="com.google.firebase.ml.vision.DEPENDENCIES"
            android:value="label" />

    </application>

</manifest>

Step 5: Working with the activity_main.xml file

Navigate to the app > res > layout > activity_main.xml and add the below code to that file. Below is the code for the activity_main.xml file. 

XML

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <!--image view for displaying our image-->
    <ImageView
        android:id="@+id/image"
        android:layout_width="match_parent"
        android:layout_height="300dp"
        android:layout_alignParentTop="true"
        android:layout_centerHorizontal="true"
        android:layout_marginTop="10dp"
        android:scaleType="centerCrop" />

    <LinearLayout
        android:id="@+id/idLLButtons"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_below="@id/image"
        android:orientation="horizontal">

        <!--button for capturing our image-->
        <Button
            android:id="@+id/snapbtn"
            android:layout_width="0dp"
            android:layout_height="wrap_content"
            android:layout_margin="5dp"
            android:layout_marginTop="30dp"
            android:layout_weight="1"
            android:text="SNAP"
            android:textSize="25dp"
            android:textStyle="bold" />

        <!--button for detecting the objects-->
        <Button
            android:id="@+id/labelBtn"
            android:layout_width="0dp"
            android:layout_height="wrap_content"
            android:layout_margin="5dp"
            android:layout_marginTop="30dp"
            android:layout_weight="1"
            android:text="Label"
            android:textSize="25dp"
            android:textStyle="bold" />

    </LinearLayout>

    <!--recycler view for displaying the list of objects-->
    <androidx.recyclerview.widget.RecyclerView
        android:id="@+id/idRVResults"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_below="@id/idLLButtons" />

</RelativeLayout>

Step 6: Creating a modal class for storing our data

Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name your class as DataModal and add the below code to it. Comments are added inside the code to understand the code in more detail.

Java

public class DataModal {

    // variables for our
    // string and confidence.
    private String result;
    private float confidence;

    // constructor
    public DataModal(String result, float confidence) {
        this.result = result;
        this.confidence = confidence;
    }

    // getter and setter methods
    public float getConfidence() {
        return confidence;
    }

    public void setConfidence(float confidence) {
        this.confidence = confidence;
    }

    public String getResult() {
        return result;
    }

    public void setResult(String result) {
        this.result = result;
    }

}

Step 7: Creating a layout file for displaying our recycler view items

Navigate to the app > res > layout > Right-click on it > New > layout resource file and name it as result_rv_item and add below code to it. 

XML

<?xml version="1.0" encoding="utf-8"?>
<androidx.cardview.widget.CardView
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:layout_margin="5dp"
    android:elevation="8dp"
    app:cardCornerRadius="8dp">

    <RelativeLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_margin="3dp">

        <!--text view for our result-->
        <TextView
            android:id="@+id/idTVResult"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:padding="3dp"
            android:text="Result"
            android:textColor="@color/black" />

        <!--text view for our confidence-->
        <TextView
            android:id="@+id/idTVConfidence"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:layout_below="@id/idTVResult"
            android:padding="3dp"
            android:text="Confidence"
            android:textColor="@color/black" />

    </RelativeLayout>

</androidx.cardview.widget.CardView>

Step 8: Creating an adapter class for our RecyclerView

Navigate to the app > java > your app’s package name > Right-click on it > New > Java class and name it as resultRVAdapter and add the below code to it. Comments are added inside the code to understand the code in more detail.

Java

import android.content.Context;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;

import androidx.annotation.NonNull;
import androidx.recyclerview.widget.RecyclerView;

import java.util.ArrayList;

public class resultRVAdapter extends RecyclerView.Adapter<resultRVAdapter.ViewHolder> {

    // arraylist for storing our data and context
    private ArrayList<DataModal> dataModalArrayList;
    private Context context;

    // constructor for our variables
    public resultRVAdapter(ArrayList<DataModal> dataModalArrayList, Context context) {
        this.dataModalArrayList = dataModalArrayList;
        this.context = context;
    }

    @NonNull
    @Override
    public ViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {
        // inside on create view holder method we are inflating our layout file which we created.
        View view = LayoutInflater.from(parent.getContext()).inflate(R.layout.result_rv_item, parent, false);
        return new ViewHolder(view);
    }

    @Override
    public void onBindViewHolder(@NonNull ViewHolder holder, int position) {
        // inside on bind view holder method we are setting
        // data to each item of recycler view.
        DataModal modal = dataModalArrayList.get(position);
        holder.resultTV.setText(modal.getResult());
        holder.confidenceTV.setText("" + modal.getConfidence());
    }

    @Override
    public int getItemCount() {
        // returning the size of array list.
        return dataModalArrayList.size();
    }

    public class ViewHolder extends RecyclerView.ViewHolder {
        // creating variables for our text view.
        private TextView resultTV, confidenceTV;

        public ViewHolder(@NonNull View itemView) {
            super(itemView);
            // initializing our views with their ids.
            resultTV = itemView.findViewById(R.id.idTVResult);
            confidenceTV = itemView.findViewById(R.id.idTVConfidence);
        }
    }
}

Step 9: Working with the MainActivity.java file

Go to the MainActivity.java file and refer to the following code. Below is the code for the MainActivity.java file. Comments are added inside the code to understand the code in more detail.

Java

import android.content.Intent;
import android.graphics.Bitmap;
import android.os.Bundle;
import android.provider.MediaStore;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.Toast;

import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.recyclerview.widget.LinearLayoutManager;
import androidx.recyclerview.widget.RecyclerView;

import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.firebase.ml.vision.FirebaseVision;
import com.google.firebase.ml.vision.common.FirebaseVisionImage;
import com.google.firebase.ml.vision.label.FirebaseVisionImageLabel;
import com.google.firebase.ml.vision.label.FirebaseVisionImageLabeler;

import java.util.ArrayList;
import java.util.List;

public class MainActivity extends AppCompatActivity {

    // variables for our image view, image bitmap,
    // buttons, recycler view, adapter and array list.
    private ImageView img;
    private Button snap, labelBtn;
    private Bitmap imageBitmap;
    private RecyclerView resultRV;
    private resultRVAdapter resultRvAdapter;
    private ArrayList<DataModal> dataModalArrayList;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        // initializing all our variables for views
        img = (ImageView) findViewById(R.id.image);
        snap = (Button) findViewById(R.id.snapbtn);
        labelBtn = findViewById(R.id.labelBtn);
        resultRV = findViewById(R.id.idRVResults);

        // initializing our array list
        dataModalArrayList = new ArrayList<>();

        // initializing our adapter class.
        resultRvAdapter = new resultRVAdapter(dataModalArrayList, MainActivity.this);

        // layout manager for our recycler view.
        LinearLayoutManager manager = new LinearLayoutManager(MainActivity.this);

        // on below line we are setting layout manager
        // and adapter to our recycler view.
        resultRV.setLayoutManager(manager);
        resultRV.setAdapter(resultRvAdapter);

        // adding on click listener for our label button.
        labelBtn.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                // calling method
                // to label images.
                labelImage();
            }
        });
        // adding on click listener for our snap button.
        snap.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                // calling a method
                // to capture an image.
                dispatchTakePictureIntent();
            }
        });
    }

    static final int REQUEST_IMAGE_CAPTURE = 1;

    private void dispatchTakePictureIntent() {
        // inside this method we are calling an implicit intent to capture an image.
        Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        if (takePictureIntent.resolveActivity(getPackageManager()) != null) {
            // calling a start activity for result when image is captured.
            startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        super.onActivityResult(requestCode, resultCode, data);
        // inside on activity result method we are setting
        // our image to our image view from bitmap.
        if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {
            Bundle extras = data.getExtras();
            imageBitmap = (Bitmap) extras.get("data");
            // on below line we are setting our
            // bitmap to our image view.
            img.setImageBitmap(imageBitmap);
        }
    }

    private void labelImage() {

        // inside the label image method we are calling a
        // firebase vision image and passing our image bitmap to it.
        FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(imageBitmap);

        // on below line we are creating a labeler for our image bitmap
        // and creating a variable for our firebase vision image labeler.
        FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler();

        // calling a method to process an image and adding on success listener method to it.
        labeler.processImage(image).addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
            @Override
            public void onSuccess(List<FirebaseVisionImageLabel> firebaseVisionImageLabels) {

                  // inside on success method we are running a loop to get the data from our list.
                for (FirebaseVisionImageLabel label : firebaseVisionImageLabels) {

                    // on below line we are getting text from our list.
                    String text = label.getText();

                    // on below line we are getting its entity id
                    String entityId = label.getEntityId();

                    // on below line we are getting the
                    // confidence level of our modal.
                    float confidence = label.getConfidence();

                    // after getting all data we are passing it to our array list.
                    dataModalArrayList.add(new DataModal(text, confidence));

                    // after adding a new data we are notifying
                    // our adapter that data has been updated.
                    resultRvAdapter.notifyDataSetChanged();
                }
            }
        }).addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // error handling for on failure method
                Toast.makeText(MainActivity.this, "Fail to get data..", Toast.LENGTH_SHORT).show();
            }
        });
    }
}

Now run your app and see the output of the app. 

Output:

Note: If you are facing the following NDK at ~/Library/Android/sdk/ndk-bundle did not have a source.properties file error then please refer to this article.