Augmented Faces with ARCore in Android
Augmented Faces permit the application to naturally distinguish various regions of an individual’s face, and utilize those areas to overlay resources, for example, surfaces and models in a way that appropriately matches the contours and regions of an individual face. ARCore is a stage for building Augmented reality applications on Android. Augmented Face is a subsystem of ARCore that permits your application to:
- Naturally, recognize various areas of any individual’s identified face, and utilize those regions to overlay resources, for example, surfaces and models in a way that appropriately matches the contours and regions of an individual face.
- Utilize the 468-point face mesh that is given by ARCore to apply a custom texture over a distinguished face.
For example, we can create effects like animated masks, glasses, virtual hats, perform skin retouching, or the next Snapchat App.
How Does it All Work?
Augmented faces don’t require uncommon or special hardware, such as a depth sensor. Rather, it uses the phone’s camera and machine learning to provide three snippets of data:
- Generates a Face Mesh: a 468 points dense 3D face mesh, which allows you to pan detailed textures that accurately follow facial moments.
- Recognizes the Pose: points on a person’s face, anchored based on the generated Face Mesh, which is useful for placing effects on or close to the temple and nose.
- Overlays and position textures and 3D models based on the face mesh generated and recognized regions.
How is ARCore Able to Provide a 3D face Mesh from a 2D Image without any Depth Hardware?
It uses machine learning models that are built on top of the TensorFlow Lite platform to achieve this and the crossing pipeline is optimized to run on the device in real-time. It uses a technique called transfer learning wherein we train the neural network for two objectives, one, to predict 3D vertices and to predict 2D contours. To predict 3D vertices, we train it with a synthetic 3D data set and use this neural network as a starting point for the next stage of training.
In this next stage, it uses the annotated data set, annotated real-world data set to train the model for 2D contour prediction. The resulting network not only predicts 3D vertices from a synthetic data set but can also perform well from 2D images. To make sure the solution works for everyone ARCore developers train the network with geographically diverse data sets so that it works for all types of faces, wider faces, taller faces, and all types of skin colors.
And to enable these complex algorithms on mobile devices, we have multiple adaptive algorithms built into the ARCore. These algorithms sense dynamically how much time it has taken to process previous images and adjust accordingly various parameters of the pipeline. It uses multiple ML models, one optimized for higher quality and one optimized for higher performance when computing the resources is really challenging. It also adjusts pipeline parameters such as inference rates so that it skips a few images, and instead replace that with interpolation data. With all these techniques, what you get is a full-frame rate experience for your user. So it provides face mesh and region poses at full camera frame rate while handling all these techniques internal to the ARCore.
Identifying an Augmented Face Mesh
To appropriately overlay textures and 3D models on an identified face, ARCore provides detected regions and an augmented face mesh. This mesh is a virtual depiction of the face and comprises the vertices, facial regions, and the focal point of the user’s head. At the point when a user’s face is identified by the camera, ARCore performs the following steps to generate the augmented face mesh, as well as center and region poses:
- It distinguishes the center pose and a face mesh.
- The center pose, situated behind the nose, is the actual center point of the user’s head (in other words, inside the skull).
- The face mesh comprises of many vertices that make up the face and is characterized relative to the center pose.
- The AugmentedFace class utilizes the face mesh and center pose to distinguish face regions present on the client’s face. These regions are:
- Right brow (RIGHT_FOREHEAD)
- Left temple (LEFT_FOREHEAD)
- Tip of the nose (NOSE_TIP)
The Face mesh, center pose, and face region poses are utilized by AugmentedFace APIs as positioning points and regions to place the resources in your app.
- Trackable: A Trackable is an interface which can be followed by ARCore and something from which Anchors can be connected to.
- Anchor: It describes a fixed location and orientation in the real world. To stay at a fixed location in physical space, the numerical description of this position will update as ARCore’s understanding of the space improves. Anchors are hashable and may for example be used as keys in HashMaps.
- Pose: At the point when you need to state wherein the scene you need to put the object and you need to specify the location in terms of the scene’s coordinates. The Pose is the means by which you state this.
- Session: Deals with the AR framework state and handles the session lifecycle. This class offers the primary passage to the ARCore API. This class permits the user to make a session, configure it, start or stop it and, above all, receive frames that allow access to the camera image and device pose.
- Textures: Textures are especially helpful for Augmented Faces. This permits you to make a light overlay that lines up with the locales of the identified face(s) to add to your experience.
- ArFragment: ARCore utilizes an ArFragment that provides a lot of features, for example, plane finding, permission handling, and camera set up. You can utilize the fragment legitimately in your activity, however at whatever point you need custom features, for example, Augmented Faces, you should extend the ArFragment and set the proper settings. This fragment is the layer that conceals all the compound stuff (like OpenGL, rendering models, etc) and gives high-level APIs to load and render 3D models.
- ModelRenderable: ModelRenderable renders a 3D Model by attaching it to a Node.
- Sceneform SDK: Sceneform SDK is another library for Android that empowers the quick creation and mix of AR experiences in your application. It joins ARCore and an amazing physically-based 3D renderer.
We are going to create Snapchat, Instagram, and TikTok like face filters. A sample GIF is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using the Java language.
Step 1: Create a New Project
To create a new project in Android Studio please refer to How to Create/Start a New Project in Android Studio. Note that select Java as the programming language.
Step 2: Adding the assets file used in this example
Add any 3D model in sampledata/models folder. We can do this by creating a new folder in the project file directory or directly from the Android Studio. The allowed 3D model extensions are .fbx, .OBJ, .glTF. There are many free models available on the internet. You can visit, here or more. You can download the assets used in this example from here. Please refer to this article to create a raw folder in android studio. Then just copy and paste the fox_face.sfb file to the raw folder. Similarly, copy and paste the fox_face_mesh_texture.png file to the drawable folder.
Step 3: Adding dependencies to the build.gradle(:app) file
Add the following dependencies to the build.gradle(:app) file.
// Provides ARCore Session and related resources.
// Provides ArFragment, and other UX resources.
// Alternatively, use ArSceneView without the UX dependency.
Add the following code snippet to the build.grdale file. This is required(only once) to convert .fbx asset into .sfb and save that in the raw folder. Or you can add them by yourself as done in step 2.
// required(only once) to convert .fbx asset into .sfb
// and save that in raw folder
Step 4: Adding dependencies to the build.gradle(:project) file
Add the following dependencies to the build.gradle(:project) file.
// Add Sceneform plugin classpath to Project
// level build.gradle file
Step 5: Working with the AndroidManifest.xml file
Add the following line to the AndroidManifest.xml file.
// Both “AR Optional” and “AR Required” apps require CAMERA permission.
<uses-permission android:name=”android.permission.CAMERA” />
// Indicates that app requires ARCore (“AR Required”). Ensures app is only
// visible in the Google Play Store on devices that support ARCore.
// For “AR Optional” apps remove this line. →
<uses-feature android:name=”android.hardware.camera.ar” android:required=”true”/>
// Indicates that app requires ARCore (“AR Required”). Causes Google
// Play Store to download and install ARCore along with the app.
// For an “AR Optional” app, specify “optional” instead of “required”.
<meta-data android:name=”com.google.ar.core” android:value=”required” />
Below is the complete code for the AndroidManifest.xml file.
Step 6: Modify the activity_main.xml file
We have added a fragment to the activity_main.xml file. Below is the code for the activity_main.xml file.
Please add your package name to this attribute.
Step 7: Create a new Java class
Create a new class and name the file as CustomArFragment that extends ArFragment. Below is the code for the CustomArFragment.java file.
Step 8: Modify the MainActivity.java file
Below is the code for the MainActivity.java file. Comments are added inside the code to understand the code in more detail.
Output: Run on a Physical Device
Github Project Link: https://github.com/raghavtilak/AugmentedFaces
Limitations of Ar Core
- Augmented Faces only works with the Front Camera.
- Not all devices support ARCore. There’s still a small fraction of devices which doesn’t come with AR Core support. You can check out the list of ARCore supported devices at https://developers.google.com/ar/discover/supported-devices.
- For AR Optional App minSdkVersion should be 14 and for AR Required App minSdkVersion should be 24.
- If your app falls in the category of AR Required app then the device using it should have AR Core installed on it.
- Prior to making a Session, it must be verified beforehand that ARCore is installed and up to date. If ARCore isn’t installed, then session creation might fail and any later installation or upgrade of ARCore would require restarting of the app, and might cause the app to be killed.
- The orientation of the face mesh is different for Unreal, Android and Unity.
- Calling Trackable.createAnchor(Pose) would result in an IllegalStateException because Augmented Faces supports only front-facing (selfie) camera, and does not support attaching anchors.