After you
train your own model using AutoML Vision Edge
,
you can use it in your app to label images.
There are two ways to integrate models trained from AutoML Vision Edge: You can
bundle the model by putting it inside your app’s asset folder, or you can
dynamically download it from Firebase.
Model bundling options
|
Bundled in your app
|
- The model is part of your app's APK
- The model is available immediately, even when the Android device is offline
- No need for a Firebase project
|
Hosted with Firebase
|
|
Before you begin
Add the dependencies for the ML Kit Android libraries to your module's
app-level gradle file, which is usually
app/build.gradle
:
For bundling a model with your app:
dependencies {
// ...
// Image labeling feature with bundled automl model
implementation 'com.google.mlkit:image-labeling-custom:16.3.1'
}
For dynamically downloading a model from Firebase, add the
linkFirebase
dependency:
dependencies {
// ...
// Image labeling feature with automl model downloaded
// from firebase
implementation 'com.google.mlkit:image-labeling-custom:16.3.1'
implementation 'com.google.mlkit:linkfirebase:16.1.0'
}
If you want to download a model
, make sure you
add Firebase to your Android project
,
if you have not already done so. This is not required when you bundle the model.
1. Load the model
To bundle the model with your app:
Extract the model and its metadata from the zip archive you downloaded
from Firebase console. We recommend you use the files as you downloaded
them, without modification (including the file names).
Include your model and its metadata files in your app package:
- If you don't have an assets folder in your project, create one by
right-clicking the
app/
folder, then clicking
New > Folder > Assets Folder
.
- Create a sub-folder under the assets folder to contain the model
files.
- Copy the files
model.tflite
,
dict.txt
, and
manifest.json
to the sub-folder (all three files must be in
the same folder).
Add the following to your app's
build.gradle
file to ensure
Gradle doesn’t compress the model file when building the app:
android {
// ...
aaptOptions {
noCompress "tflite"
}
}
The model file will be included in the app package and available to ML Kit
as a raw asset.
Create
LocalModel
object, specifying the path to the model manifest
file:
Java
AutoMLImageLabelerLocalModel localModel =
new AutoMLImageLabelerLocalModel.Builder()
.setAssetFilePath("manifest.json")
// or .setAbsoluteFilePath(absolute file path to manifest file)
.build();
Kotlin
val localModel = LocalModel.Builder()
.setAssetManifestFilePath("manifest.json")
// or .setAbsoluteManifestFilePath(absolute file path to manifest file)
.build()
To use the remotely-hosted model, create a
CustomRemoteModel
object,
specifying the name you assigned the model when you published it:
Java
// Specify the name you assigned in the Firebase console.
FirebaseModelSource firebaseModelSource =
new FirebaseModelSource.Builder("your_model_name").build();
CustomRemoteModel remoteModel =
new CustomRemoteModel.Builder(firebaseModelSource).build();
Kotlin
// Specify the name you assigned in the Firebase console.
val firebaseModelSource = FirebaseModelSource.Builder("your_model_name")
.build()
val remoteModel = CustomRemoteModel.Builder(firebaseModelSource).build()
Then, start the model download task, specifying the conditions under which
you want to allow downloading. If the model isn't on the device, or if a newer
version of the model is available, the task will asynchronously download the
model from Firebase:
Java
DownloadConditions downloadConditions = new DownloadConditions.Builder()
.requireWifi()
.build();
RemoteModelManager.getInstance().download(remoteModel, downloadConditions)
.addOnSuccessListener(new OnSuccessListener<Void>() {
@Override
public void onSuccess(@NonNull Task<Void> task) {
// Success.
}
});
Kotlin
val downloadConditions = DownloadConditions.Builder()
.requireWifi()
.build()
RemoteModelManager.getInstance().download(remoteModel, downloadConditions)
.addOnSuccessListener {
// Success.
}
Many apps start the download task in their initialization code, but you
can do so at any point before you need to use the model.
Create an image labeler from your model
After you configure your model sources, create a
ImageLabeler
object from one
of them.
If you only have a locally-bundled model, just create a labeler from your
CustomImageLabelerOptions
object and configure the confidence score
threshold you want to require (see
Evaluate your model
):
Java
CustomImageLabelerOptions customImageLabelerOptions = new CustomImageLabelerOptions.Builder(localModel)
.setConfidenceThreshold(0.0f) // Evaluate your model in the Cloud console
// to determine an appropriate value.
.build();
ImageLabeler labeler = ImageLabeling.getClient(customImageLabelerOptions);
Kotlin
val customImageLabelerOptions = CustomImageLabelerOptions.Builder(localModel)
.setConfidenceThreshold(0.0f) // Evaluate your model in the Cloud console
// to determine an appropriate value.
.build()
val labeler = ImageLabeling.getClient(customImageLabelerOptions)
If you have a remotely-hosted model, you will have to check that it has been
downloaded before you run it. You can check the status of the model download
task using the model manager's
isModelDownloaded()
method.
Although you only have to confirm this before running the labeler, if you
have both a remotely-hosted model and a locally-bundled model, it might make
sense to perform this check when instantiating the image labeler: create a
labeler from the remote model if it's been downloaded, and from the local
model otherwise.
Java
RemoteModelManager.getInstance().isModelDownloaded(remoteModel)
.addOnSuccessListener(new OnSuccessListener<Boolean>() {
@Override
public void onSuccess(Boolean isDownloaded) {
CustomImageLabelerOptions.Builder optionsBuilder;
if (isDownloaded) {
optionsBuilder = new CustomImageLabelerOptions.Builder(remoteModel);
} else {
optionsBuilder = new CustomImageLabelerOptions.Builder(localModel);
}
CustomImageLabelerOptions options = optionsBuilder
.setConfidenceThreshold(0.0f) // Evaluate your model in the Cloud console
// to determine an appropriate threshold.
.build();
ImageLabeler labeler = ImageLabeling.getClient(options);
}
});
Kotlin
RemoteModelManager.getInstance().isModelDownloaded(remoteModel)
.addOnSuccessListener { isDownloaded ->
val optionsBuilder =
if (isDownloaded) {
CustomImageLabelerOptions.Builder(remoteModel)
} else {
CustomImageLabelerOptions.Builder(localModel)
}
// Evaluate your model in the Cloud console to determine an appropriate threshold.
val options = optionsBuilder.setConfidenceThreshold(0.0f).build()
val labeler = ImageLabeling.getClient(options)
}
If you only have a remotely-hosted model, you should disable model-related
functionality—for example, grey-out or hide part of your UI—until
you confirm the model has been downloaded. You can do so by attaching a listener
to the model manager's
download()
method:
Java
RemoteModelManager.getInstance().download(remoteModel, conditions)
.addOnSuccessListener(new OnSuccessListener<Void>() {
@Override
public void onSuccess(Void v) {
// Download complete. Depending on your app, you could enable
// the ML feature, or switch from the local model to the remote
// model, etc.
}
});
Kotlin
RemoteModelManager.getInstance().download(remoteModel, conditions)
.addOnSuccessListener {
// Download complete. Depending on your app, you could enable the ML
// feature, or switch from the local model to the remote model, etc.
}
Then, for each image you want to label, create an
InputImage
object from your image. The image labeler runs fastest when you use a
Bitmap
or, if you use the camera2 API, a YUV_420_888
media.Image
, which are
recommended when possible.
You can create an
InputImage
from different sources, each is explained below.
To create an
InputImage
object from a
media.Image
object, such as when you capture an image from a
device's camera, pass the
media.Image
object and the image's
rotation to
InputImage.fromMediaImage()
.
If you use the
CameraX
library, the
OnImageCapturedListener
and
ImageAnalysis.Analyzer
classes calculate the rotation value
for you.
Kotlin+KTX
private class YourImageAnalyzer : ImageAnalysis.Analyzer {
override fun analyze(imageProxy: ImageProxy?) {
val mediaImage = imageProxy?.image
if (mediaImage != null) {
val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
// Pass image to an ML Kit Vision API
// ...
}
}
}
Java
private class YourAnalyzer implements ImageAnalysis.Analyzer {
@Override
public void analyze(ImageProxy imageProxy) {
if (imageProxy == null || imageProxy.getImage() == null) {
return;
}
Image mediaImage = imageProxy.getImage();
InputImage image =
InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees);
// Pass image to an ML Kit Vision API
// ...
}
}
If you don't use a camera library that gives you the image's rotation degree, you
can calculate it from the device's rotation degree and the orientation of camera
sensor in the device:
Kotlin+KTX
private val ORIENTATIONS = SparseIntArray()
init {
ORIENTATIONS.append(Surface.ROTATION_0, 90)
ORIENTATIONS.append(Surface.ROTATION_90, 0)
ORIENTATIONS.append(Surface.ROTATION_180, 270)
ORIENTATIONS.append(Surface.ROTATION_270, 180)
}
/**
* Get the angle by which an image must be rotated given the device's current
* orientation.
*/
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
@Throws(CameraAccessException::class)
private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
// Get the device's current rotation relative to its "native" orientation.
// Then, from the ORIENTATIONS table, look up the angle the image must be
// rotated to compensate for the device's rotation.
val deviceRotation = activity.windowManager.defaultDisplay.rotation
var rotationCompensation = ORIENTATIONS.get(deviceRotation)
// On most devices, the sensor orientation is 90 degrees, but for some
// devices it is 270 degrees. For devices with a sensor orientation of
// 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
val sensorOrientation = cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.SENSOR_ORIENTATION)!!
rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
// Return the corresponding FirebaseVisionImageMetadata rotation value.
val result: Int
when (rotationCompensation) {
0 -> result = FirebaseVisionImageMetadata.ROTATION_0
90 -> result = FirebaseVisionImageMetadata.ROTATION_90
180 -> result = FirebaseVisionImageMetadata.ROTATION_180
270 -> result = FirebaseVisionImageMetadata.ROTATION_270
else -> {
result = FirebaseVisionImageMetadata.ROTATION_0
Log.e(TAG, "Bad rotation value: $rotationCompensation")
}
}
return result
}
Java
private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
static {
ORIENTATIONS.append(Surface.ROTATION_0, 90);
ORIENTATIONS.append(Surface.ROTATION_90, 0);
ORIENTATIONS.append(Surface.ROTATION_180, 270);
ORIENTATIONS.append(Surface.ROTATION_270, 180);
}
/**
* Get the angle by which an image must be rotated given the device's current
* orientation.
*/
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
private int getRotationCompensation(String cameraId, Activity activity, Context context)
throws CameraAccessException {
// Get the device's current rotation relative to its "native" orientation.
// Then, from the ORIENTATIONS table, look up the angle the image must be
// rotated to compensate for the device's rotation.
int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
int rotationCompensation = ORIENTATIONS.get(deviceRotation);
// On most devices, the sensor orientation is 90 degrees, but for some
// devices it is 270 degrees. For devices with a sensor orientation of
// 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
int sensorOrientation = cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.SENSOR_ORIENTATION);
rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
// Return the corresponding FirebaseVisionImageMetadata rotation value.
int result;
switch (rotationCompensation) {
case 0:
result = FirebaseVisionImageMetadata.ROTATION_0;
break;
case 90:
result = FirebaseVisionImageMetadata.ROTATION_90;
break;
case 180:
result = FirebaseVisionImageMetadata.ROTATION_180;
break;
case 270:
result = FirebaseVisionImageMetadata.ROTATION_270;
break;
default:
result = FirebaseVisionImageMetadata.ROTATION_0;
Log.e(TAG, "Bad rotation value: " + rotationCompensation);
}
return result;
}
Then, pass the
media.Image
object and the
rotation degree value to
InputImage.fromMediaImage()
:
Kotlin+KTX
val image = InputImage.fromMediaImage(mediaImage, rotation)
Java
InputImage image = InputImage.fromMediaImage(mediaImage, rotation);
Using a file URI
To create an
InputImage
object from a file URI, pass
the app context and file URI to
InputImage.fromFilePath()
. This is useful when you
use an
ACTION_GET_CONTENT
intent to prompt the user to select
an image from their gallery app.
Kotlin+KTX
val image: InputImage
try {
image = InputImage.fromFilePath(context, uri)
} catch (e: IOException) {
e.printStackTrace()
}
Java
InputImage image;
try {
image = InputImage.fromFilePath(context, uri);
} catch (IOException e) {
e.printStackTrace();
}
Using a
ByteBuffer
or
ByteArray
To create an
InputImage
object from a
ByteBuffer
or a
ByteArray
, first calculate the image
rotation degree as previously described for
media.Image
input.
Then, create the
InputImage
object with the buffer or array, together with image's
height, width, color encoding format, and rotation degree:
Kotlin+KTX
val image = InputImage.fromByteBuffer(
byteBuffer,
/* image width */ 480,
/* image height */ 360,
rotationDegrees,
InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
)
Java
InputImage image = InputImage.fromByteBuffer(byteBuffer,
/* image width */ 480,
/* image height */ 360,
rotationDegrees,
InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
);
Using a
Bitmap
To create an
InputImage
object from a
Bitmap
object, make the following declaration:
Kotlin+KTX
val image = InputImage.fromBitmap(bitmap, 0)
Java
InputImage image = InputImage.fromBitmap(bitmap, rotationDegree);
The image is represented by a
Bitmap
object together with rotation degrees.
3. Run the image labeler
To label objects in an image, pass the
image
object to the
ImageLabeler
's
process()
method.
Java
labeler.process(image)
.addOnSuccessListener(new OnSuccessListener<List<ImageLabel>>() {
@Override
public void onSuccess(List<ImageLabel> labels) {
// Task completed successfully
// ...
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
// ...
}
});
Kotlin
labeler.process(image)
.addOnSuccessListener { labels ->
// Task completed successfully
// ...
}
.addOnFailureListener { e ->
// Task failed with an exception
// ...
}
If the image labeling operation succeeds, a list of
ImageLabel
objects is passed to the success listener. Each
ImageLabel
object represents
something that was labeled in the image. You can get each label's text
description, the confidence score of the match and the index of the match.
For example:
Java
for (ImageLabel label : labels) {
String text = label.getText();
float confidence = label.getConfidence();
int index = label.getIndex();
}
Kotlin
for (label in labels) {
val text = label.text
val confidence = label.confidence
val index = label.index
}
If you want to label images in a real-time application, follow these
guidelines to achieve the best framerates:
- Throttle calls to the image labeler. If a new video frame becomes
available while the image labeler is running, drop the frame. See the
VisionProcessorBase
class in the quickstart sample app for an example.
- If you are using the output of the image labeler to overlay graphics on
the input image, first get the result, then render the image
and overlay in a single step. By doing so, you render to the display surface
only once for each input frame. See the
CameraSourcePreview
and
GraphicOverlay
classes in the quickstart sample app for an
example.
-
If you use the Camera2 API, capture images in
ImageFormat.YUV_420_888
format.
If you use the older Camera API, capture images in
ImageFormat.NV21
format.