MediaPipeTasksVision Framework Reference

InteractiveSegmenter

class InteractiveSegmenter : NSObject

@brief Class that performs interactive segmentation on images.

Users can represent user interaction through RegionOfInterest, which gives a hint to InteractiveSegmenter to perform segmentation focusing on the given region of interest.

The API expects a TFLite model with mandatory TFLite Model Metadata.

Input tensor: (kTfLiteUInt8/kTfLiteFloat32)

  • image input of size [batch x height x width x channels].
  • batch inference is not supported (batch is required to be 1).
  • RGB and greyscale inputs are supported (channels is required to be 1 or 3).
  • if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization. Output tensors: (kTfLiteUInt8/kTfLiteFloat32)
  • list of segmented masks.
  • if output_type is CATEGORY_MASK, uint8 Image, Image vector of size 1.
  • if output_type is CONFIDENCE_MASK, float32 Image list of size channels.
  • batch is always 1.

An example of such model can be found at: https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2

  • Get the category label list of the InteractiveSegmenter can recognize. For CATEGORY_MASK type, the index in the category mask corresponds to the category in the label list. For CONFIDENCE_MASK type, the output mask list at index corresponds to the category in the label list. If there is no labelmap provided in the model file, empty array is returned.

    Declaration

    Swift

    var labels: [String] { get }

  • Creates a new instance of InteractiveSegmenter from an absolute path to a TensorFlow Lite model file stored locally on the device and the default InteractiveSegmenterOptions.

    Declaration

    Swift

    convenience init(modelPath: String) throws

    Parameters

    modelPath

    An absolute path to a TensorFlow Lite model file stored locally on the device.

    Return Value

    A new instance of InteractiveSegmenter with the given model path. nil if there is an error in initializing the interactive segmenter.

  • Creates a new instance of InteractiveSegmenter from the given InteractiveSegmenterOptions.

    Declaration

    Swift

    init(options: InteractiveSegmenterOptions) throws

    Parameters

    options

    The options of type InteractiveSegmenterOptions to use for configuring the InteractiveSegmenter.

    Return Value

    A new instance of InteractiveSegmenter with the given options. nil if there is an error in initializing the interactive segmenter.

  • Performs segmentation on the provided MPPImage using the specified user’s region of interest. Rotation will be applied according to the orientation property of the provided MPImage.

    This method supports interactive segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of .image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(image: MPImage, regionOfInterest: RegionOfInterest) throws -> InteractiveSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.

    Return Value

    An InteractiveSegmenterResult that contains the segmented masks.

  • Performs segmentation on the provided MPPImage using the specified user’s region of interest and invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns.

    Rotation will be applied according to the orientation property of the provided MPImage.

    This method supports interactive segmentation of RGBA images. If your MPImage has a source type of .pixelBuffer or .sampleBuffer, the underlying pixel buffer must use kCVPixelFormatType_32BGRA as its pixel format.

    If your MPImage has a source type of image ensure that the color space is RGB with an Alpha channel.

    Declaration

    Swift

    func segment(image: MPImage, regionOfInterest: RegionOfInterest) async throws -> InteractiveSegmenterResult

    Parameters

    image

    The MPImage on which segmentation is to be performed.
    completionHandler

    A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional InteractiveSegmenterResult that contains the segmented masks if the segmentation was successful and an optional error populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block.

  • Undocumented

  • Undocumented