`viam.services.vision`

Module Contents

Classes

`VisModelType`	str(object='') -> str
`VisModelConfig`
`VisionServiceClient`	Connect to the Vision service, which allows you to access various computer vision algorithms

class viam.services.vision.VisModelType[source]

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

DETECTOR_TF_LITE = tflite_detector

DETECTOR_TENSORFLOW = tf_detector

DETECTOR_COLOR = color_detector

CLASSIFIER_TFLITE = tflite_classifier

CLASSIFIER_TENSORFLOW = tf_classifier

DETECTOR_SEGMENTER = detector_segmenter

RADIUS_CLUSTERING_SEGMENTER = radius_clustering_segmenter

class viam.services.vision.VisModelConfig[source]

name :str

type :VisModelType

parameters :Mapping[str, Any]

class viam.services.vision.VisionServiceClient(name: str, channel: grpclib.client.Channel)[source]

Connect to the Vision service, which allows you to access various computer vision algorithms (like detection, segmentation, tracking, etc) that usually only require a camera or image input.

SERVICE_TYPE = vision

async get_detector_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[str][source]

Get the list of detectors currently registered in the service.

Returns: The detector names
Return type: List[str]

async add_detector(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a new detector to the service. Returns nothing if successful, and an error if not. Registers a new detector just as if you had put it in the original “register_models” field in the robot config. Available types and their parameters can be found in the vision service documentation.

Parameters: config (VisModelConfig) – The configuration of the detector to add.

async remove_detector(detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the detector with the given name from the service. Returns nothing if successful.

Parameters: detector_name (str) – The name of the detector to remove

async get_detections_from_camera(camera_name: str, detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[viam.proto.service.vision.Detection][source]

Get a list of detections in the next image given a camera and a detector

Parameters

camera_name (str) – The name of the camera to use for detection
detector_name (str) – The name of the detector to use for detection

Returns

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type

List[Detection]

async get_detections(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[viam.proto.service.vision.Detection][source]

Get a list of detections in the given image using the specified detector

Parameters

image (Image) – The image to get detections from
detector_name (str) – The name of the detector to use for detection

Returns

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type

List[Detection]

async get_classifier_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[str][source]

Get the list of classifiers currently registered to the service

Returns: The list of classifier names
Return type: List[str]

async add_classifier(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a classifier to the service.

Parameters: config (VisModelConfig) – The configuration of the classifier

async remove_classifier(classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the classifier with the given name from the service. Returns nothing if successful.

Parameters: classifier_name (str) – The name of the classifier to remove

async get_classifications_from_camera(camera_name: str, classifier_name: str, count: int, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[viam.proto.service.vision.Classification][source]

Get a list of classifications in the next image given a camera and a classifier

Parameters

camera_name (str) – The name of the camera to use for detection
classifier_name (str) – The name of the classifier to use for classification
count (int) – The number of classifications desired

Returns

The list of Classifications

Return type

List[Classification]

async get_classifications(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[viam.proto.service.vision.Classification][source]

Get a list of detections in the given image using the specified detector

Parameters

image (Image) – The image to get detections from
classifier_name (str) – The name of the detector to use for detection

Returns

The list of Classifications

Return type

List[Classification]

async get_segmenter_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[str][source]

Get the list of segmenters currently registered in the service.

Returns: The segmenter names
Return type: List[str]

async add_segmenter(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Add a segmenter to the service

Parameters: config (VisModelConfig) – The configuration of the segmenter

async remove_segmenter(segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]

Remove the segmenter with the given name from the service. Returns nothing if successful.

Parameters: segmenter_name (str) – The name of the segmenter to remove

async get_model_parameters_schema(model_type: VisModelType, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → Mapping[str, Union[str, int, float, bool, Sequence, Mapping]][source]

Get the parameters needed to add a model to the vision registry.

Parameters: model_type (VisModelType) – The name of model
Returns: A dictionary representing the parameters as JSONSchema
Return type: Mapping[str, str | int | float | bool | Sequence | Mapping]

async get_object_point_clouds(camera_name: str, segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) → List[viam.proto.common.PointCloudObject][source]

Returns a list of the 3D point cloud objects and associated metadata in the latest picture obtained from the specified 3D camera (using the specified segmenter).

To deserialize the returned information into a numpy array, use the Open3D library.

import numpy as np
import open3d as o3d

object_point_clouds = await vision.get_object_point_clouds(camera_name, segmenter_name)

# write the first object point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
    f.write(object_point_clouds[0].point_cloud)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)

Parameters

camera_name (str) – The name of the camera
segmenter_name (str) – The name of the segmenter

Returns

The pointcloud objects with metadata

Return type

List[PointCloudObject]

viam.services.vision

Module Contents

Classes

`viam.services.vision`