You can use Firebase ML to label objects recognized in an image. See the
overview
for information about this API's
features.
Before you begin
If you have not already added Firebase to your app, do so by following the
steps in the
getting started guide
.
Use Swift Package Manager to install and manage Firebase dependencies.
- In Xcode, with your app project open, navigate to
File > Add Packages
.
- When prompted, add the Firebase Apple platforms SDK repository:
https://github.com/firebase/firebase-ios-sdk.git
- Choose the Firebase ML library.
- Add the
-ObjC
flag to the
Other Linker Flags
section of your target's build settings.
-
When finished, Xcode will automatically begin resolving and downloading your
dependencies in the background.
Next, perform some in-app setup:
- In your app, import Firebase:
Swift
import FirebaseMLModelDownloader
Objective-C
@import FirebaseMLModelDownloader;
-
If you have not already enabled Cloud-based APIs for your project, do so
now:
- Open the
Firebase ML
APIs page
of the Firebase console.
-
If you have not already upgraded your project to the Blaze pricing plan, click
Upgrade
to do so. (You will be prompted to upgrade only if your
project isn't on the Blaze plan.)
Only Blaze-level projects can use Cloud-based APIs.
- If Cloud-based APIs aren't already enabled, click
Enable Cloud-based
APIs
.
Now you are ready to label images.
Create a
VisionImage
object using a
UIImage
or a
CMSampleBufferRef
.
To use a
UIImage
:
- If necessary, rotate the image so that its
imageOrientation
property is
.up
.
- Create a
VisionImage
object using the correctly-rotated
UIImage
. Do not specify any rotation metadata—the default
value,
.topLeft
, must be used.
Swift
let image = VisionImage(image: uiImage)
Objective-C
FIRVisionImage *image = [[FIRVisionImage alloc] initWithImage:uiImage];
To use a
CMSampleBufferRef
:
-
Create a
VisionImageMetadata
object that specifies the
orientation of the image data contained in the
CMSampleBufferRef
buffer.
To get the image orientation:
Swift
func imageOrientation(
deviceOrientation: UIDeviceOrientation,
cameraPosition: AVCaptureDevice.Position
) -> VisionDetectorImageOrientation {
switch deviceOrientation {
case .portrait:
return cameraPosition == .front ? .leftTop : .rightTop
case .landscapeLeft:
return cameraPosition == .front ? .bottomLeft : .topLeft
case .portraitUpsideDown:
return cameraPosition == .front ? .rightBottom : .leftBottom
case .landscapeRight:
return cameraPosition == .front ? .topRight : .bottomRight
case .faceDown, .faceUp, .unknown:
return .leftTop
}
}
Objective-C
- (FIRVisionDetectorImageOrientation)
imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
cameraPosition:(AVCaptureDevicePosition)cameraPosition {
switch (deviceOrientation) {
case UIDeviceOrientationPortrait:
if (cameraPosition == AVCaptureDevicePositionFront) {
return FIRVisionDetectorImageOrientationLeftTop;
} else {
return FIRVisionDetectorImageOrientationRightTop;
}
case UIDeviceOrientationLandscapeLeft:
if (cameraPosition == AVCaptureDevicePositionFront) {
return FIRVisionDetectorImageOrientationBottomLeft;
} else {
return FIRVisionDetectorImageOrientationTopLeft;
}
case UIDeviceOrientationPortraitUpsideDown:
if (cameraPosition == AVCaptureDevicePositionFront) {
return FIRVisionDetectorImageOrientationRightBottom;
} else {
return FIRVisionDetectorImageOrientationLeftBottom;
}
case UIDeviceOrientationLandscapeRight:
if (cameraPosition == AVCaptureDevicePositionFront) {
return FIRVisionDetectorImageOrientationTopRight;
} else {
return FIRVisionDetectorImageOrientationBottomRight;
}
default:
return FIRVisionDetectorImageOrientationTopLeft;
}
}
Then, create the metadata object:
Swift
let cameraPosition = AVCaptureDevice.Position.back // Set to the capture device you used.
let metadata = VisionImageMetadata()
metadata.orientation = imageOrientation(
deviceOrientation: UIDevice.current.orientation,
cameraPosition: cameraPosition
)
Objective-C
FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
AVCaptureDevicePosition cameraPosition =
AVCaptureDevicePositionBack; // Set to the capture device you used.
metadata.orientation =
[self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
cameraPosition:cameraPosition];
- Create a
VisionImage
object using the
CMSampleBufferRef
object and the rotation metadata:
Swift
let image = VisionImage(buffer: sampleBuffer)
image.metadata = metadata
Objective-C
FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:sampleBuffer];
image.metadata = metadata;
To label objects in an image, pass the
VisionImage
object to the
VisionImageLabeler
's
processImage()
method.
First, get an instance of
VisionImageLabeler
:
Swift
let labeler = Vision.vision().cloudImageLabeler()
// Or, to set the minimum confidence required:
// let options = VisionCloudImageLabelerOptions()
// options.confidenceThreshold = 0.7
// let labeler = Vision.vision().cloudImageLabeler(options: options)
Objective-C
FIRVisionImageLabeler *labeler = [[FIRVision vision] cloudImageLabeler];
// Or, to set the minimum confidence required:
// FIRVisionCloudImageLabelerOptions *options =
// [[FIRVisionCloudImageLabelerOptions alloc] init];
// options.confidenceThreshold = 0.7;
// FIRVisionImageLabeler *labeler =
// [[FIRVision vision] cloudImageLabelerWithOptions:options];
Then, pass the image to the
processImage()
method:
Swift
labeler.process(image) { labels, error in
guard error == nil, let labels = labels else { return }
// Task succeeded.
// ...
}
Objective-C
[labeler processImage:image
completion:^(NSArray<FIRVisionImageLabel *> *_Nullable labels,
NSError *_Nullable error) {
if (error != nil) { return; }
// Task succeeded.
// ...
}];
If image labeling succeeds, an array of
VisionImageLabel
objects will be passed to the completion handler. From each object, you can get
information about a feature recognized in the image.
For example:
Swift
for label in labels {
let labelText = label.text
let entityId = label.entityID
let confidence = label.confidence
}
Objective-C
for (FIRVisionImageLabel *label in labels) {
NSString *labelText = label.text;
NSString *entityId = label.entityID;
NSNumber *confidence = label.confidence;
}
Next steps