ImageSegmentationService class
Separates frames of the video feed into classified regions, for example body/background. Used for background subtraction/virtual green-screen effects.
Signature:
export declare class ImageSegmentationService extends ExternalObject
Extends: ExternalObject
Remarks
For most use cases it is not recommended to use the ImageSegmentationService to apply a green screen effect to a component. Instead, apply a green screen filter to the component's media source (and display the component above the module view with LayoutCustomConfig.displayAboveModule, if needed) as that generally provides better performance.
Methods
| Method | Modifiers | Description |
|---|---|---|
| setTargetFps(fps) | Sets the system to process at a target FPS. Will only reach the maximum real fps supported by system & device capture. | |
| start() | Starts the image segmentation so that it begins to send frames of data | |
| stop() | Stop the image segmentation | |
| subscribeToComposite(desiredWidth, desiredHeight, scaleMode) | Subscribe to the texture feed for an alpha-blended background-subtracted texture. Each subject update will be a TextureData that can be drawn to a canvas. | |
| subscribeToMask(desiredWidth, desiredHeight, scaleMode) | Subscribe to the texture feed for the mask only. Each subject update will be a TextureData that can be drawn to a canvas. |
ImageSegmentationService.setTargetFps() method
Sets the system to process at a target FPS. Will only reach the maximum real fps supported by system & device capture.
Signature:
setTargetFps(fps: number): void;
Parameters
| Parameter | Type | Description |
|---|---|---|
| fps | number | The number of frames per second you would like results at. Must be a positive integer. |
Returns:
void
ImageSegmentationService.stop() method
Stop the image segmentation
Signature:
stop(): void;
Returns:
void
ImageSegmentationService.start() method
Starts the image segmentation so that it begins to send frames of data
Signature:
start(): void;
Returns:
void
ImageSegmentationService.subscribeToComposite() method
Subscribe to the texture feed for an alpha-blended background-subtracted texture. Each subject update will be a TextureData that can be drawn to a canvas.
Signature:
subscribeToComposite(desiredWidth: number, desiredHeight: number, scaleMode?: ImageSegmentationScaleMode): Subject<TextureData>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| desiredWidth | number | The desired maximum width of the returned texture. The returned texture is guaranteed to be this width if you use exact scale mode. |
| desiredHeight | number | The desired maximum height of the returned texture. The returned texture is guaranteed to be this height if you use exact scale mode. |
| scaleMode | ImageSegmentationScaleMode | (Optional) How the desiredWidth and desiredHeight should be used to determine the output texture size. See ImageSegmentationScaleMode for the possible options. |
Returns:
Remarks
The frame will be a full RGBA image with the background subtracted. Opaque pixels are those classified as body/foreground and transparent pixels are classified as background.
Request small textures (192x256 or 384x512) and scale them up, as this requires significant memory. Note that returned texture size is not guaranteed to match requested texture size.
To receive textures in the proper aspect ratio, you can calculate the aspect ratio ahead of time by using CameraComponent.getCaptureResolution() for camera components or AssetManager.getMetadata() for videos; or you can use the ImageSegmentationScaleMode.FitInside scale mode, which will give you a texture that fits inside your requested dimensions but has a correct aspect ratio.
When using ImageSegmentationScaleMode.FitInside be careful not to create the ImageData instance until you receive a TextureData for the first time. Then you can create it using the matching dimensions from TextureData.width and TextureData.height. You can also use TextureData.toImageData() to create one of the proper size.
Note that the same TextureData instance is returned with every update of the subscription.
ImageSegmentationService.subscribeToMask() method
Subscribe to the texture feed for the mask only. Each subject update will be a TextureData that can be drawn to a canvas.
Signature:
subscribeToMask(desiredWidth: number, desiredHeight: number, scaleMode?: ImageSegmentationScaleMode): Subject<TextureData>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| desiredWidth | number | The desired maximum width of the returned texture. The returned texture is guaranteed to be this width if you use exact scale mode. |
| desiredHeight | number | The desired maximum height of the returned texture. The returned texture is guaranteed to be this height if you use exact scale mode. |
| scaleMode | ImageSegmentationScaleMode | (Optional) How the desiredWidth and desiredHeight should be used to determine the output texture size. See ImageSegmentationScaleMode for the possible options. |
Returns:
Remarks
The "mask" is an RGBA image where the value of the red channel represents the background. A fully red pixel is background and a fully black pixel is foreground.
Request small textures (192x256 or 384x512) and scale them up, as this requires significant memory. Note that returned texture size is not guaranteed to match requested texture size.
To receive textures in the proper aspect ratio, you can calculate the aspect ratio ahead of time by using CameraComponent.getCaptureResolution() for camera components or AssetManager.getMetadata() for videos; or you can use the ImageSegmentationScaleMode.FitInside scale mode, which will give you a texture that fits inside your requested dimensions but has a correct aspect ratio.
When using ImageSegmentationScaleMode.FitInside be careful not to create the ImageData instance until you receive a TextureData for the first time. Then you can create it using the matching dimensions from TextureData.width and TextureData.height. You can also use TextureData.toImageData() to create one of the proper size.
Note that the same TextureData instance is returned with every update of the subscription.
Updated about 1 year ago