MediaPipeTasksVision Framework Reference

ImageSegmenter

class ImageSegmenter : NSObject

@brief Class that performs segmentation on images.

The API expects a TFLite model with mandatory TFLite Model Metadata.

  • Get the category label list of the ImageSegmenter can recognize. For CATEGORY_MASK type, the index in the category mask corresponds to the category in the label list. For CONFIDENCE_MASK type, the output mask list at index corresponds to the category in the label list. If there is no labelmap provided in the model file, empty array is returned.

    Declaration

    Swift

    var labels: [String] { get }

  • Creates a new instance of ImageSegmenter from an absolute path to a TensorFlow Lite model file stored locally on the device and the default ImageSegmenterOptions.

    Declaration

    Swift

    convenience init(modelPath: String) throws

    Parameters

    modelPath

    An absolute path to a TensorFlow Lite model file stored locally on the device.

    Return Value

    A new instance of ImageSegmenter with the given model path. nil if there is an error in initializing the image segmenter.

  • Creates a new instance of ImageSegmenter from the given ImageSegmenterOptions.

    Declaration

    Swift

    init(options: ImageSegmenterOptions) throws

    Parameters

    options

    The options of type ImageSegmenterOptions to use for configuring the ImageSegmenter.

    Return Value

    A new instance of ImageSegmenter with the given options. nil if there is an error in initializing the image segmenter.

  • Performs segmentation on the provided MPPImage using the whole image as region of interest. Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageSegmenter is created with running mode, image.

    This method supports segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(image: MPImage) throws -> ImageSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.

    Return Value

    An ImageSegmenterResult that contains the segmented masks.

  • Performs segmentation on the provided MPPImage using the whole image as region of interest and invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns.

    Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageSegmenter is created with running mode, image.

    This method supports segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(image: MPImage) async throws -> ImageSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.
    completionHandler

    A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional ImageSegmenterResult that contains the segmented masks if the segmentation was successful and an optional error populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block.

  • Performs segmentation on the provided video frame of type MPImage using the whole image as region of interest.

    Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageSegmenter is created with video.

    This method supports segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(videoFrame image: MPImage, timestampInMilliseconds: Int) throws -> ImageSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.
    timestampInMilliseconds

    The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.

    Return Value

    An ImageSegmenterResult that contains a the segmented masks.

  • Performs segmentation on the provided video frame of type MPImage using the whole image as region of interest invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns.

    Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageSegmenter is created with running mode, video.

    This method supports segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(videoFrame image: MPImage, timestampInMilliseconds: Int) async throws -> ImageSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.
    timestampInMilliseconds

    The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.
    completionHandler

    A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional ImageSegmenterResult that contains the segmented masks if the segmentation was successful and an optional error only populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block.

  • Sends live stream image data of type MPImage to perform segmentation using the whole image as region of interest.

    Rotation will be applied according to the orientation property of the provided MPImage. Only use this method when the ImageSegmenter is created with running mode, liveStream.

    The object which needs to be continuously notified of the available results of image segmentation must confirm to ImageSegmenterLiveStreamDelegate protocol and implement the imageSegmenter(_:didFinishSegmentationWithResult:timestampInMilliseconds:error:) delegate method.

    It’s required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the segmenter. The input timestamps must be monotonically increasing.

    This method supports segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If the input MPImage has a source type of image ensure that the color space is RGB with an Alpha channel.

    If this method is used for classifying live camera frames using AVFoundation, ensure that you request AVCaptureVideoDataOutput to output frames in kCMPixelFormat_32BGRA using its videoSettings property.

    Declaration

    Swift

    func segmentAsync(image: MPImage, timestampInMilliseconds: Int) throws

    Parameters

    image

    A live stream image data of type MPImage on which segmentation is to be performed.
    timestampInMilliseconds

    The timestamp (in milliseconds) which indicates when the input image is sent to the segmenter. The input timestamps must be monotonically increasing.

    Return Value

    YES if the image was sent to the task successfully, otherwise NO.

  • Undocumented

  • Undocumented