viam.services.vision

Module Contents

Classes

VisModelType

str(object='') -> str

VisModelConfig

VisionServiceClient

Connect to the Vision service, which allows you to access various computer vision algorithms

class viam.services.vision.VisModelType[source]

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

DETECTOR_TF_LITE = tflite_detector
DETECTOR_TENSORFLOW = tf_detector
DETECTOR_COLOR = color_detector
CLASSIFIER_TFLITE = tflite_classifier
CLASSIFIER_TENSORFLOW = tf_classifier
DETECTOR_SEGMENTER = detector_segmenter
RADIUS_CLUSTERING_SEGMENTER = radius_clustering_segmenter
class viam.services.vision.VisModelConfig[source]
name :str
type :VisModelType
parameters :Mapping[str, Any]
class viam.services.vision.VisionServiceClient(name: str, channel: grpclib.client.Channel)[source]

Connect to the Vision service, which allows you to access various computer vision algorithms (like detection, segmentation, tracking, etc) that usually only require a camera or image input.

SERVICE_TYPE = vision
async get_detector_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str][source]

Get the list of detectors currently registered in the service.

Returns

The detector names

Return type

List[str]

async add_detector(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a new detector to the service. Returns nothing if successful, and an error if not. Registers a new detector just as if you had put it in the original “register_models” field in the robot config. Available types and their parameters can be found in the vision service documentation.

Parameters

config (VisModelConfig) – The configuration of the detector to add.

async remove_detector(detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the detector with the given name from the service. Returns nothing if successful.

Parameters

detector_name (str) – The name of the detector to remove

async get_detections_from_camera(camera_name: str, detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Detection][source]

Get a list of detections in the next image given a camera and a detector

Parameters
  • camera_name (str) – The name of the camera to use for detection

  • detector_name (str) – The name of the detector to use for detection

Returns

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type

List[Detection]

async get_detections(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Detection][source]

Get a list of detections in the given image using the specified detector

Parameters
  • image (Image) – The image to get detections from

  • detector_name (str) – The name of the detector to use for detection

Returns

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type

List[Detection]

async get_classifier_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str][source]

Get the list of classifiers currently registered to the service

Returns

The list of classifier names

Return type

List[str]

async add_classifier(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a classifier to the service.

Parameters

config (VisModelConfig) – The configuration of the classifier

async remove_classifier(classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the classifier with the given name from the service. Returns nothing if successful.

Parameters

classifier_name (str) – The name of the classifier to remove

async get_classifications_from_camera(camera_name: str, classifier_name: str, count: int, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Classification][source]

Get a list of classifications in the next image given a camera and a classifier

Parameters
  • camera_name (str) – The name of the camera to use for detection

  • classifier_name (str) – The name of the classifier to use for classification

  • count (int) – The number of classifications desired

Returns

The list of Classifications

Return type

List[Classification]

async get_classifications(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Classification][source]

Get a list of detections in the given image using the specified detector

Parameters
  • image (Image) – The image to get detections from

  • classifier_name (str) – The name of the detector to use for detection

Returns

The list of Classifications

Return type

List[Classification]

async get_segmenter_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str][source]

Get the list of segmenters currently registered in the service.

Returns

The segmenter names

Return type

List[str]

async add_segmenter(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a segmenter to the service

Parameters

config (VisModelConfig) – The configuration of the segmenter

async remove_segmenter(segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the segmenter with the given name from the service. Returns nothing if successful.

Parameters

segmenter_name (str) – The name of the segmenter to remove

async get_model_parameters_schema(model_type: VisModelType, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) Mapping[str, Union[str, int, float, bool, Sequence, Mapping]][source]

Get the parameters needed to add a model to the vision registry.

Parameters

model_type (VisModelType) – The name of model

Returns

A dictionary representing the parameters as JSONSchema

Return type

Mapping[str, str | int | float | bool | Sequence | Mapping]

async get_object_point_clouds(camera_name: str, segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.common.PointCloudObject][source]

Returns a list of the 3D point cloud objects and associated metadata in the latest picture obtained from the specified 3D camera (using the specified segmenter).

To deserialize the returned information into a numpy array, use the Open3D library.

import numpy as np
import open3d as o3d

object_point_clouds = await vision.get_object_point_clouds(camera_name, segmenter_name)

# write the first object point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
    f.write(object_point_clouds[0].point_cloud)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)
Parameters
  • camera_name (str) – The name of the camera

  • segmenter_name (str) – The name of the segmenter

Returns

The pointcloud objects with metadata

Return type

List[PointCloudObject]