MediaPipeTasksVision Framework Reference

MPPImageEmbedder


@interface MPPImageEmbedder : NSObject

@brief Performs embedding extraction on images.

The API expects a TFLite model with optional, but strongly recommended, TFLite Model Metadata..

The API supports models with one image input tensor and one or more output tensors. To be more specific, here are the requirements.

Input image tensor (kTfLiteUInt8/kTfLiteFloat32)

  • image input of size [batch x height x width x channels].
  • batch inference is not supported (batch is required to be 1).
  • only RGB inputs are supported (channels is required to be 3).
  • if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.

At least one output tensor (kTfLiteUInt8/kTfLiteFloat32) with shape [1 x N] where N is the number of dimensions in the produced embeddings.

  • Creates a new instance of ImageEmbedder from an absolute path to a TensorFlow Lite model file stored locally on the device and the default ImageEmbedderOptions.

    Declaration

    Objective-C

    - (nullable instancetype)initWithModelPath:(nonnull NSString *)modelPath
                                     error:(NSError *_Nullable *_Nullable)error;

    Parameters

    modelPath

    An absolute path to a TensorFlow Lite model file stored locally on the device.

    Return Value

    A new instance of ImageEmbedder with the given model path. nil if there is an error in initializing the image embedder.

  • Creates a new instance of ImageEmbedder from the given ImageEmbedderOptions.

    Declaration

    Objective-C

    - (nullable instancetype)initWithOptions:
                             (nonnull MPPImageEmbedderOptions *)options
                                   error:(NSError *_Nullable *_Nullable)error;

    Parameters

    options

    The options of type ImageEmbedderOptions to use for configuring the ImageEmbedder.

    Return Value

    A new instance of ImageEmbedder with the given options. nil if there is an error in initializing the image embedder.

  • Performs embedding extraction on the provided MPImage using the whole image as region of interest. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with running mode, .image.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Objective-C

    - (nullable MPPImageEmbedderResult *)embedImage:(nonnull MPPImage *)image
                                          error:(NSError *_Nullable *_Nullable)
                                                    error;

    Parameters

    image

    The MPImage on which embedding extraction is to be performed.

    Return Value

    An ImageEmbedderResult object that contains a list of embedding extraction.

  • Performs embedding extraction on the provided MPImage cropped to the specified region of interest. Rotation will be applied on the cropped image according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with running mode, .image.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Objective-C

    - (nullable MPPImageEmbedderResult *)embedImage:(nonnull MPPImage *)image
                               regionOfInterest:(CGRect)roi
                                          error:(NSError *_Nullable *_Nullable)
                                                    error;

    Parameters

    image

    The MPImage on which embedding extraction is to be performed.
    roi

    A CGRect specifying the region of interest within the given MPImage, on which embedding extraction should be performed.

    Return Value

    An ImageEmbedderResult object that contains a list of generated image embeddings.

  • Performs embedding extraction on the provided video frame of type MPImage using the whole image as region of interest. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with running mode .video.

    It’s required to provide the video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Objective-C

    - (nullable MPPImageEmbedderResult *)
            embedVideoFrame:(nonnull MPPImage *)image
    timestampInMilliseconds:(NSInteger)timestampInMilliseconds
                      error:(NSError *_Nullable *_Nullable)error;

    Parameters

    image

    The MPImage on which embedding extraction is to be performed.
    timestampInMilliseconds

    The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.

    Return Value

    An ImageEmbedderResult object that contains a list of generated image embeddings.

  • Performs embedding extraction on the provided video frame of type MPImage cropped to the specified region of interest. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with .video.

    It’s required to provide the video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Objective-C

    - (nullable MPPImageEmbedderResult *)
            embedVideoFrame:(nonnull MPPImage *)image
    timestampInMilliseconds:(NSInteger)timestampInMilliseconds
           regionOfInterest:(CGRect)roi
                      error:(NSError *_Nullable *_Nullable)error;

    Parameters

    image

    A live stream image data of type MPImage on which embedding extraction is to be performed.
    timestampInMilliseconds

    The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.
    roi

    A CGRect specifying the region of interest within the video frame of type MPImage, on which embedding extraction should be performed.

    Return Value

    An ImageEmbedderResult object that contains a list of generated image embeddings.

  • Sends live stream image data of type MPImage to perform embedding extraction using the whole image as region of interest. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with running mode .liveStream.

    The object which needs to be continuously notified of the available results of image embedding extraction must confirm to ImageEmbedderLiveStreamDelegate protocol and implement the imageEmbedder(_:didFinishEmbeddingWithResult:timestampInMilliseconds:error:) delegate method.

    It’s required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If the input MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    If this method is used for embedding live camera frames using AVFoundation, ensure that you request AVCaptureVideoDataOutput to output frames in kCMPixelFormat_32BGRA using its videoSettings property.

    Declaration

    Objective-C

    - (BOOL)embedAsyncImage:(nonnull MPPImage *)image
    timestampInMilliseconds:(NSInteger)timestampInMilliseconds
                      error:(NSError *_Nullable *_Nullable)error;

    Parameters

    image

    A live stream image data of type MPImage on which embedding extraction is to be performed.
    timestampInMilliseconds

    The timestamp (in milliseconds) which indicates when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.

    Return Value

    true if the image was sent to the task successfully, otherwise false.

  • Sends live stream image data of type MPImage to perform embedding extraction, cropped to the specified region of interest.. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageEmbedder is created with .liveStream.

    The object which needs to be continuously notified of the available results of image embedding extraction must confirm to ImageEmbedderLiveStreamDelegate protocol and implement the imageEmbedder(_:didFinishEmbeddingWithResult:timestampInMilliseconds:error:) delegate method.

    It’s required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.

    This method supports embedding extraction on RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If the input MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    If this method is used for embedding live camera frames using AVFoundation, ensure that you request AVCaptureVideoDataOutput to output frames in kCMPixelFormat_32BGRA using its videoSettings property.

    Declaration

    Objective-C

    - (BOOL)embedAsyncImage:(nonnull MPPImage *)image
    timestampInMilliseconds:(NSInteger)timestampInMilliseconds
           regionOfInterest:(CGRect)roi
                      error:(NSError *_Nullable *_Nullable)error;

    Parameters

    image

    A live stream image data of type MPImage on which embedding extraction is to be performed.
    timestampInMilliseconds

    The timestamp (in milliseconds) which indicates when the input image is sent to the image embedder. The input timestamps must be monotonically increasing.
    roi

    A CGRect specifying the region of interest within the given live stream image data of type MPImage, on which embedding extraction should be performed.

    Return Value

    true if the image was sent to the task successfully, otherwise false.

  • Undocumented

    Declaration

    Objective-C

    - (instancetype)init NS_UNAVAILABLE;

  • Utility function to computecosine similarity between two MPPEmbedding objects.

    Declaration

    Objective-C

    + (nullable NSNumber *)
    cosineSimilarityBetweenEmbedding1:(nonnull MPPEmbedding *)embedding1
                        andEmbedding2:(nonnull MPPEmbedding *)embedding2
                                error:(NSError *_Nullable *_Nullable)error;

    Parameters

    embedding1

    One of the two MPPEmbeddings between whom cosine similarity is to be computed.
    embedding2

    One of the two MPPEmbeddings between whom cosine similarity is to be computed.
    error

    An optional error parameter populated when there is an error in calculating cosine similarity between two embeddings.

    Return Value

    An NSNumber which holds the cosine similarity of type double.

  • Undocumented

    Declaration

    Objective-C

    + (instancetype)new NS_UNAVAILABLE;