MPImage class

The wrapper class for MediaPipe Image objects.

Images are stored as ImageData, ImageBitmap or WebGLTexture objects. You can convert the underlying type to any other type by passing the desired type to getAs...(). As type conversions can be expensive, it is recommended to limit these conversions. You can verify what underlying types are already available by invoking has...().

Images that are returned from a MediaPipe Tasks are owned by by the underlying C++ Task. If you need to extend the lifetime of these objects, you can invoke the clone() method. To free up the resources obtained during any clone or type conversion operation, it is important to invoke close() on the MPImage instance.

Converting to and from ImageBitmap requires that the MediaPipe task is initialized with an OffscreenCanvas. As we require WebGL2 support, this places some limitations on Browser support as outlined here: https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas/getContext

Signature:

export declare class MPImage 

Properties

Methods

MPImage.canvas

Returns the canvas element that the image is bound to.

Signature:

readonly canvas: HTMLCanvasElement | OffscreenCanvas | undefined;

MPImage.height

Returns the height of the image.

Signature:

readonly height: number;

MPImage.width

Returns the width of the image.

Signature:

readonly width: number;

MPImage.clone()

Creates a copy of the resources stored in this MPImage. You can invoke this method to extend the lifetime of an image returned by a MediaPipe Task. Note that performance critical applications should aim to only use the MPImage within the MediaPipe Task callback so that copies can be avoided.

Signature:

clone(): MPImage;

Returns:

MPImage

MPImage.close()

Frees up any resources owned by this MPImage instance.

Note that this method does not free images that are owned by the C++ Task, as these are freed automatically once you leave the MediaPipe callback. Additionally, some shared state is freed only once you invoke the Task's close() method.

Signature:

close(): void;

Returns:

void

MPImage.getAsImageBitmap()

Returns the underlying image as an ImageBitmap. Note that conversions to ImageBitmap are expensive, especially if the data currently resides on CPU.

Processing with ImageBitmaps requires that the MediaPipe Task was initialized with an OffscreenCanvas with WebGL2 support. See https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas/getContext for a list of supported platforms.

The current image as an ImageBitmap object.

Signature:

getAsImageBitmap(): ImageBitmap;

Returns:

ImageBitmap

MPImage.getAsImageData()

Returns the underlying image as an ImageData object. Note that this involves an expensive GPU to CPU transfer if the current image is only available as an ImageBitmap or WebGLTexture.

The current image as an ImageData object.

Signature:

getAsImageData(): ImageData;

Returns:

ImageData

MPImage.getAsWebGLTexture()

Returns the underlying image as a WebGLTexture object. Note that this involves a CPU to GPU transfer if the current image is only available as an ImageData object. The returned texture is bound to the current canvas (see .canvas).

The current image as a WebGLTexture.

Signature:

getAsWebGLTexture(): WebGLTexture;

Returns:

WebGLTexture

MPImage.hasImageBitmap()

Returns whether this MPImage contains a mask of type ImageBitmap.

Signature:

hasImageBitmap(): boolean;

Returns:

boolean

MPImage.hasImageData()

Returns whether this MPImage contains a mask of type ImageData.

Signature:

hasImageData(): boolean;

Returns:

boolean

MPImage.hasWebGLTexture()

Returns whether this MPImage contains a mask of type WebGLTexture.

Signature:

hasWebGLTexture(): boolean;

Returns:

boolean