viam.services.vision.client

Attributes

ValueTypes

Types that can be encoded into a protobuf Value

Exceptions

ViamError

Base Viam Error

Classes

CameraMimeType

str(object='') -> str

ViamImage

A native implementation of an image.

DoCommandRequest

DoCommandRequest represents a generic DoCommand input

DoCommandResponse

DoCommandResponse represents a generic DoCommand output

PointCloudObject

PointCloudObject contains an image in bytes with point cloud data of all of the objects captured by a given observer as well as a

CaptureAllFromCameraRequest

Abstract base class for protocol messages.

CaptureAllFromCameraResponse

Abstract base class for protocol messages.

Classification

the general form of the output from a classifier

Detection

Abstract base class for protocol messages.

GetClassificationsFromCameraRequest

Abstract base class for protocol messages.

GetClassificationsFromCameraResponse

Abstract base class for protocol messages.

GetClassificationsRequest

Abstract base class for protocol messages.

GetClassificationsResponse

Abstract base class for protocol messages.

GetDetectionsFromCameraRequest

Abstract base class for protocol messages.

GetDetectionsFromCameraResponse

Abstract base class for protocol messages.

GetDetectionsRequest

Abstract base class for protocol messages.

GetDetectionsResponse

Abstract base class for protocol messages.

GetObjectPointCloudsRequest

Abstract base class for protocol messages.

GetObjectPointCloudsResponse

Abstract base class for protocol messages.

GetPropertiesRequest

Abstract base class for protocol messages.

GetPropertiesResponse

Abstract base class for protocol messages.

VisionServiceStub

ReconfigurableResourceRPCClientBase

A base RPC client that can reset its channel.

CaptureAllResult

CaptureAllResult represents the collection of things that you have requested from the

Vision

Vision represents a Vision service.

VisionClient

Connect to the Vision service, which allows you to access various computer vision algorithms

Functions

dict_to_struct(→ google.protobuf.struct_pb2.Struct)

struct_to_dict(→ Dict[str, ValueTypes])

Module Contents

exception viam.services.vision.client.ViamError[source]

Bases: Exception

Base Viam Error

class viam.services.vision.client.CameraMimeType[source]

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

VIAM_RGBA = 'image/vnd.viam.rgba'
VIAM_RAW_DEPTH = 'image/vnd.viam.dep'
JPEG = 'image/jpeg'
PNG = 'image/png'
PCD = 'pointcloud/pcd'
classmethod from_string(value: str) typing_extensions.Self[source]
classmethod from_proto(format: viam.proto.component.camera.Format.ValueType) CameraMimeType[source]

Returns the mimetype from a proto enum.

Parameters:

format (Format.ValueType) – The mimetype in a proto enum.

Returns:

The mimetype.

Return type:

Self

to_proto() viam.proto.component.camera.Format.ValueType[source]

Returns the mimetype in a proto enum.

Returns:

The mimetype in a proto enum.

Return type:

Format.ValueType

class viam.services.vision.client.ViamImage(data: bytes, mime_type: CameraMimeType)[source]

A native implementation of an image.

Provides the raw data and the mime type.

property data: bytes

The raw bytes of the image

property mime_type: CameraMimeType

The mime type of the image

property width: int | None

The width of the image

property height: int | None

The height of the image

bytes_to_depth_array() List[List[int]][source]

Decode the data of an image that has the custom depth MIME type image/vnd.viam.dep into a standard representation.

Raises:

NotSupportedError – Raised if the image is not of MIME type image/vnd.viam.dep.

Returns:

The standard representation of the image.

Return type:

List[List[int]]

class viam.services.vision.client.DoCommandRequest(*, name: str = ..., command: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

DoCommandRequest represents a generic DoCommand input

name: str
property command: google.protobuf.struct_pb2.Struct
HasField(field_name: Literal['command', b'command']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.DoCommandResponse(*, result: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

DoCommandResponse represents a generic DoCommand output

property result: google.protobuf.struct_pb2.Struct
HasField(field_name: Literal['result', b'result']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.PointCloudObject(*, point_cloud: bytes = ..., geometries: global___GeometriesInFrame | None = ...)

Bases: google.protobuf.message.Message

PointCloudObject contains an image in bytes with point cloud data of all of the objects captured by a given observer as well as a repeated list of geometries which respresents the center point and geometry of each of the objects within the point cloud

point_cloud: bytes

image frame expressed in bytes

property geometries: global___GeometriesInFrame

volume of a given geometry

HasField(field_name: Literal['geometries', b'geometries']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.CaptureAllFromCameraRequest(*, name: str = ..., camera_name: str = ..., return_image: bool = ..., return_classifications: bool = ..., return_detections: bool = ..., return_object_point_clouds: bool = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

camera_name: str

name of camera source to use as input

return_image: bool

whether or not including the image in the response

return_classifications: bool

whether or not including classifications in the response

return_detections: bool

whether or not including detections in the response

return_object_point_clouds: bool

whether or not including pcd in the response

property extra: google.protobuf.struct_pb2.Struct
HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.CaptureAllFromCameraResponse(*, image: viam.gen.component.camera.v1.camera_pb2.Image | None = ..., detections: collections.abc.Iterable[global___Detection] | None = ..., classifications: collections.abc.Iterable[global___Classification] | None = ..., objects: collections.abc.Iterable[viam.gen.common.v1.common_pb2.PointCloudObject] | None = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

property image: viam.gen.component.camera.v1.camera_pb2.Image
property detections: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Detection]
property classifications: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Classification]
property objects: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[viam.gen.common.v1.common_pb2.PointCloudObject]
property extra: google.protobuf.struct_pb2.Struct
HasField(field_name: Literal['extra', b'extra', 'image', b'image']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.Classification(*, class_name: str = ..., confidence: float = ...)

Bases: google.protobuf.message.Message

the general form of the output from a classifier

class_name: str

the class name

confidence: float

the confidence score of the classification

class viam.services.vision.client.Detection(*, x_min: int | None = ..., y_min: int | None = ..., x_max: int | None = ..., y_max: int | None = ..., confidence: float = ..., class_name: str = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

x_min: int

the four corners of the box

y_min: int
x_max: int
y_max: int
confidence: float

the confidence of the detection

class_name: str

label associated with the detected object

HasField(field_name: Literal['_x_max', b'_x_max', '_x_min', b'_x_min', '_y_max', b'_y_max', '_y_min', b'_y_min', 'x_max', b'x_max', 'x_min', b'x_min', 'y_max', b'y_max', 'y_min', b'y_min']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

WhichOneof(oneof_group: Literal['_x_max', b'_x_max']) Literal['x_max'] | None
WhichOneof(oneof_group: Literal['_x_min', b'_x_min']) Literal['x_min'] | None
WhichOneof(oneof_group: Literal['_y_max', b'_y_max']) Literal['y_max'] | None
WhichOneof(oneof_group: Literal['_y_min', b'_y_min']) Literal['y_min'] | None

Returns the name of the field that is set inside a oneof group.

If no field is set, returns None.

Parameters:

oneof_group (str) – the name of the oneof group to check.

Returns:

The name of the group that is set, or None.

Return type:

str or None

Raises:

ValueError – no group with the given name exists

class viam.services.vision.client.GetClassificationsFromCameraRequest(*, name: str = ..., camera_name: str = ..., n: int = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

camera_name: str

the image encoded as bytes

n: int

the number of classifications desired

property extra: google.protobuf.struct_pb2.Struct

Additional arguments to the method

HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetClassificationsFromCameraResponse(*, classifications: collections.abc.Iterable[global___Classification] | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

property classifications: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Classification]
class viam.services.vision.client.GetClassificationsRequest(*, name: str = ..., image: bytes = ..., width: int = ..., height: int = ..., mime_type: str = ..., n: int = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

image: bytes

the image encoded as bytes

width: int

the width of the image

height: int

the height of the image

mime_type: str

the actual MIME type of image

n: int

the number of classifications desired

property extra: google.protobuf.struct_pb2.Struct

Additional arguments to the method

HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetClassificationsResponse(*, classifications: collections.abc.Iterable[global___Classification] | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

property classifications: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Classification]
class viam.services.vision.client.GetDetectionsFromCameraRequest(*, name: str = ..., camera_name: str = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

camera_name: str

name of camera source to use as input

property extra: google.protobuf.struct_pb2.Struct
HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetDetectionsFromCameraResponse(*, detections: collections.abc.Iterable[global___Detection] | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

property detections: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Detection]

the bounding boxes and labels

class viam.services.vision.client.GetDetectionsRequest(*, name: str = ..., image: bytes = ..., width: int = ..., height: int = ..., mime_type: str = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

image: bytes

the image, encoded as bytes

width: int

the width of the image

height: int

the height of the image

mime_type: str

the actual MIME type of image

property extra: google.protobuf.struct_pb2.Struct

Additional arguments to the method

HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetDetectionsResponse(*, detections: collections.abc.Iterable[global___Detection] | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

property detections: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Detection]

the bounding boxes and labels

class viam.services.vision.client.GetObjectPointCloudsRequest(*, name: str = ..., camera_name: str = ..., mime_type: str = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str
camera_name: str

Name of a camera

mime_type: str

Requested MIME type of response

property extra: google.protobuf.struct_pb2.Struct

Additional arguments to the method

HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetObjectPointCloudsResponse(*, mime_type: str = ..., objects: collections.abc.Iterable[viam.gen.common.v1.common_pb2.PointCloudObject] | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

mime_type: str

Actual MIME type of response

property objects: google.protobuf.internal.containers.RepeatedCompositeFieldContainer[viam.gen.common.v1.common_pb2.PointCloudObject]

List of objects in the scene

class viam.services.vision.client.GetPropertiesRequest(*, name: str = ..., extra: google.protobuf.struct_pb2.Struct | None = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

name: str

name of the vision service

property extra: google.protobuf.struct_pb2.Struct

Additional arguments to the method

HasField(field_name: Literal['extra', b'extra']) bool

Checks if a certain field is set for the message.

For a oneof group, checks if any field inside is set. Note that if the field_name is not defined in the message descriptor, ValueError will be raised.

Parameters:

field_name (str) – The name of the field to check for presence.

Returns:

Whether a value has been set for the named field.

Return type:

bool

Raises:

ValueError – if the field_name is not a member of this message.

class viam.services.vision.client.GetPropertiesResponse(*, classifications_supported: bool = ..., detections_supported: bool = ..., object_point_clouds_supported: bool = ...)

Bases: google.protobuf.message.Message

Abstract base class for protocol messages.

Protocol message classes are almost always generated by the protocol compiler. These generated types subclass Message and implement the methods shown below.

classifications_supported: bool

whether or not classifactions are supported by the vision service

detections_supported: bool

whether or not detections are supported by the vision service

object_point_clouds_supported: bool

whether or not 3d segmentation is supported by the vision service

class viam.services.vision.client.VisionServiceStub(channel: grpclib.client.Channel)[source]
class viam.services.vision.client.ReconfigurableResourceRPCClientBase[source]

Bases: ResourceRPCClientBase

A base RPC client that can reset its channel.

Useful if connection is lost and then regained.

reset_channel(channel: grpclib.client.Channel)[source]

Called when the RPC channel was reset. Passes in the new channel.

Parameters:

channel (Channel) – The new RPC Channel

viam.services.vision.client.ValueTypes

Types that can be encoded into a protobuf Value

viam.services.vision.client.dict_to_struct(obj: Mapping[str, ValueTypes]) google.protobuf.struct_pb2.Struct[source]
viam.services.vision.client.struct_to_dict(struct: google.protobuf.struct_pb2.Struct) Dict[str, ValueTypes][source]
class viam.services.vision.client.CaptureAllResult(image=None, classifications=None, detections=None, objects=None, extra={})[source]

CaptureAllResult represents the collection of things that you have requested from the CaptureAllFromCamera method. This is used most often for visualization purposes, since normally, returning the image on every call to a classifier/detector/etc would be costly and unnecessary. The default result for each field is None rather than the empty list to distinguish between “there was no request for the classifier/detector to return a result” vs. “the classifier/detector was requested, but there were no results”.

class viam.services.vision.client.Vision(name: str)[source]

Bases: viam.services.service_base.ServiceBase

Vision represents a Vision service.

This acts as an abstract base class for any drivers representing specific vision implementations. This cannot be used on its own. If the __init__() function is overridden, it must call the super().__init__() function.

SUBTYPE: Final
Properties: TypeAlias = GetPropertiesResponse

Properties is a class that states what features are supported on the associated vision service. Currently, these are the following properties: - classifications_supported (bool): GetClassifications and GetClassificationsFromCamera are implemented. - detections_supported (bool): GetDetections and GetDetectionsFromCamera are implemented. - object_point_clouds_supported (bool): GetObjectPointClouds is implemented.

abstract capture_all_from_camera(camera_name: str, return_image: bool = False, return_classifications: bool = False, return_detections: bool = False, return_object_point_clouds: bool = False, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) CaptureAllResult[source]
Async:

Get the next image, detections, classifications, and objects all together, given a camera name. Used for visualization.

camera_name = "cam1"

# Grab the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# capture all from the next image from the camera
result = await my_detector.capture_all_from_camera(
    camera_name,
    return_image=True,
    return_detections=True,
)
Parameters:
  • camera_name (str) – The name of the camera to use for detection

  • return_image (bool) – Ask the vision service to return the camera’s latest image

  • return_classifications (bool) – Ask the vision service to return its latest classifications

  • return_detections (bool) – Ask the vision service to return its latest detections

  • return_object_point_clouds (bool) – Ask the vision service to return its latest 3D segmentations

Returns:

A class that stores all potential returns from the vision service. It can return the image from the camera along with its associated detections, classifications, and objects, as well as any extra info the model may provide.

Return type:

vision.CaptureAllResult

abstract get_detections_from_camera(camera_name: str, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Detection][source]
Async:

Get a list of detections in the next image given a camera and a detector

camera_name = "cam1"

# Grab the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# Get detections from the next image from the camera
detections = await my_detector.get_detections_from_camera(camera_name)
Parameters:

camera_name (str) – The name of the camera to use for detection

Raises:

ViamError – Raised if given an image without a specified width and height

Returns:

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type:

List[viam.proto.service.vision.Detection]

abstract get_detections(image: viam.media.video.ViamImage, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Detection][source]
Async:

Get a list of detections in the given image using the specified detector

# Grab camera from the machine
cam1 = Camera.from_robot(robot, "cam1")

# Get the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# Get an image from the camera
img = await cam1.get_image()

# Get detections from that image
detections = await my_detector.get_detections(img)
Parameters:

image (Image | RawImage) – The image to get detections from

Raises:

ViamError – Raised if given an image without a specified width and height

Returns:

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type:

List[viam.proto.service.vision.Detection]

abstract get_classifications_from_camera(camera_name: str, count: int, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Classification][source]
Async:

Get a list of classifications in the next image given a camera and a classifier

camera_name = "cam1"

# Grab the classifier you configured on your machine
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Get the 2 classifications with the highest confidence scores from the next image from the camera
classifications = await my_classifier.get_classifications_from_camera(
    camera_name, 2)
Parameters:
  • camera_name (str) – The name of the camera to use for detection

  • count (int) – The number of classifications desired

Returns:

The list of Classifications

Return type:

List[viam.proto.service.vision.Classification]

abstract get_classifications(image: viam.media.video.ViamImage, count: int, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Classification][source]
Async:

Get a list of classifications in the given image using the specified classifier

# Grab camera from the machine
cam1 = Camera.from_robot(robot, "cam1")

# Get the classifier you configured on your machine
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Get an image from the camera
img = await cam1.get_image()

# Get the 2 classifications with the highest confidence scores
classifications = await my_classifier.get_classifications(img, 2)
Parameters:
  • image (Image | RawImage) – The image to get detections from

  • count (int) – The number of classifications desired

Returns:

The list of Classifications

Return type:

List[viam.proto.service.vision.Classification]

abstract get_object_point_clouds(camera_name: str, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.common.PointCloudObject][source]
Async:

Returns a list of the 3D point cloud objects and associated metadata in the latest picture obtained from the specified 3D camera (using the specified segmenter).

To deserialize the returned information into a numpy array, use the Open3D library.

import numpy as np
import open3d as o3d

# Grab the 3D camera from the machine
cam1 = Camera.from_robot(robot, "cam1")
# Grab the object segmenter you configured on your machine
my_segmenter = VisionClient.from_robot(robot, "my_segmenter")
# Get the objects from the camera output
objects = await my_segmenter.get_object_point_clouds(cam1)
# write the first object point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
    f.write(objects[0].point_cloud)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)
Parameters:

camera_name (str) – The name of the camera

Returns:

The pointcloud objects with metadata

Return type:

List[viam.proto.common.PointCloudObject]

abstract get_properties(*, extra: Mapping[str, Any] | None = None, timeout: float | None = None) Properties[source]
Async:

Get info about what vision methods the vision service provides. Currently returns boolean values that state whether the service implements the classification, detection, and/or 3D object segmentation methods.

::

# Grab the detector you configured on your machine my_detector = VisionClient.from_robot(robot, “my_detector”) properties = await my_detector.get_properties() properties.detections_supported # returns True properties.classifications_supported # returns False

Returns:

The properties of the vision service

Return type:

Properties

classmethod from_robot(robot: viam.robot.client.RobotClient, name: str) typing_extensions.Self

Get the service named name from the provided robot.

async def connect() -> ViamClient:
    # Replace "<API-KEY>" (including brackets) with your API key and "<API-KEY-ID>" with your API key ID
    dial_options = DialOptions.with_api_key("<API-KEY>", "<API-KEY-ID>")
    return await ViamClient.create_from_dial_options(dial_options)

async def main():
    robot = await connect()

    # Can be used with any resource, using the motion service as an example
    motion = MotionClient.from_robot(robot=robot, name="builtin")

    robot.close()
Parameters:
  • robot (RobotClient) – The robot

  • name (str) – The name of the service

Returns:

The service, if it exists on the robot

Return type:

Self

abstract do_command(command: Mapping[str, viam.utils.ValueTypes], *, timeout: float | None = None, **kwargs) Mapping[str, viam.utils.ValueTypes]
Async:

Send/receive arbitrary commands.

motion = MotionClient.from_robot(robot, "builtin")

my_command = {
  "cmnd": "dosomething",
  "someparameter": 52
}

# Can be used with any resource, using the motion service as an example
await motion.do_command(command=my_command)
Parameters:

command (Dict[str, ValueTypes]) – The command to execute

Returns:

Result of the executed command

Return type:

Dict[str, ValueTypes]

classmethod get_resource_name(name: str) viam.proto.common.ResourceName

Get the ResourceName for this Resource with the given name

# Can be used with any resource, using an arm as an example
my_arm_name = my_arm.get_resource_name("my_arm")
Parameters:

name (str) – The name of the Resource

Returns:

The ResourceName of this Resource

Return type:

ResourceName

get_operation(kwargs: Mapping[str, Any]) viam.operations.Operation

Get the Operation associated with the currently running function.

When writing custom resources, you should get the Operation by calling this function and check to see if it’s cancelled. If the Operation is cancelled, then you can perform any necessary (terminating long running tasks, cleaning up connections, etc. ).

Parameters:

kwargs (Mapping[str, Any]) – The kwargs object containing the operation

Returns:

The operation associated with this function

Return type:

viam.operations.Operation

async close()

Safely shut down the resource and prevent further use.

Close must be idempotent. Later configuration may allow a resource to be “open” again. If a resource does not want or need a close function, it is assumed that the resource does not need to return errors when future non-Close methods are called.

await component.close()
class viam.services.vision.client.VisionClient(name: str, channel: grpclib.client.Channel)[source]

Bases: viam.services.vision.vision.Vision, viam.resource.rpc_client_base.ReconfigurableResourceRPCClientBase

Connect to the Vision service, which allows you to access various computer vision algorithms (like detection, segmentation, tracking, etc) that usually only require a camera or image input.

client: viam.proto.service.vision.VisionServiceStub
async capture_all_from_camera(camera_name: str, return_image: bool = False, return_classifications: bool = False, return_detections: bool = False, return_object_point_clouds: bool = False, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) viam.services.vision.vision.CaptureAllResult[source]

Get the next image, detections, classifications, and objects all together, given a camera name. Used for visualization.

camera_name = "cam1"

# Grab the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# capture all from the next image from the camera
result = await my_detector.capture_all_from_camera(
    camera_name,
    return_image=True,
    return_detections=True,
)
Parameters:
  • camera_name (str) – The name of the camera to use for detection

  • return_image (bool) – Ask the vision service to return the camera’s latest image

  • return_classifications (bool) – Ask the vision service to return its latest classifications

  • return_detections (bool) – Ask the vision service to return its latest detections

  • return_object_point_clouds (bool) – Ask the vision service to return its latest 3D segmentations

Returns:

A class that stores all potential returns from the vision service. It can return the image from the camera along with its associated detections, classifications, and objects, as well as any extra info the model may provide.

Return type:

vision.CaptureAllResult

async get_detections_from_camera(camera_name: str, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Detection][source]

Get a list of detections in the next image given a camera and a detector

camera_name = "cam1"

# Grab the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# Get detections from the next image from the camera
detections = await my_detector.get_detections_from_camera(camera_name)
Parameters:

camera_name (str) – The name of the camera to use for detection

Raises:

ViamError – Raised if given an image without a specified width and height

Returns:

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type:

List[viam.proto.service.vision.Detection]

async get_detections(image: viam.media.video.ViamImage, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Detection][source]

Get a list of detections in the given image using the specified detector

# Grab camera from the machine
cam1 = Camera.from_robot(robot, "cam1")

# Get the detector you configured on your machine
my_detector = VisionClient.from_robot(robot, "my_detector")

# Get an image from the camera
img = await cam1.get_image()

# Get detections from that image
detections = await my_detector.get_detections(img)
Parameters:

image (Image | RawImage) – The image to get detections from

Raises:

ViamError – Raised if given an image without a specified width and height

Returns:

A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Return type:

List[viam.proto.service.vision.Detection]

async get_classifications_from_camera(camera_name: str, count: int, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Classification][source]

Get a list of classifications in the next image given a camera and a classifier

camera_name = "cam1"

# Grab the classifier you configured on your machine
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Get the 2 classifications with the highest confidence scores from the next image from the camera
classifications = await my_classifier.get_classifications_from_camera(
    camera_name, 2)
Parameters:
  • camera_name (str) – The name of the camera to use for detection

  • count (int) – The number of classifications desired

Returns:

The list of Classifications

Return type:

List[viam.proto.service.vision.Classification]

async get_classifications(image: viam.media.video.ViamImage, count: int, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.service.vision.Classification][source]

Get a list of classifications in the given image using the specified classifier

# Grab camera from the machine
cam1 = Camera.from_robot(robot, "cam1")

# Get the classifier you configured on your machine
my_classifier = VisionClient.from_robot(robot, "my_classifier")

# Get an image from the camera
img = await cam1.get_image()

# Get the 2 classifications with the highest confidence scores
classifications = await my_classifier.get_classifications(img, 2)
Parameters:
  • image (Image | RawImage) – The image to get detections from

  • count (int) – The number of classifications desired

Returns:

The list of Classifications

Return type:

List[viam.proto.service.vision.Classification]

async get_object_point_clouds(camera_name: str, *, extra: Mapping[str, Any] | None = None, timeout: float | None = None) List[viam.proto.common.PointCloudObject][source]

Returns a list of the 3D point cloud objects and associated metadata in the latest picture obtained from the specified 3D camera (using the specified segmenter).

To deserialize the returned information into a numpy array, use the Open3D library.

import numpy as np
import open3d as o3d

# Grab the 3D camera from the machine
cam1 = Camera.from_robot(robot, "cam1")
# Grab the object segmenter you configured on your machine
my_segmenter = VisionClient.from_robot(robot, "my_segmenter")
# Get the objects from the camera output
objects = await my_segmenter.get_object_point_clouds(cam1)
# write the first object point cloud into a temporary file
with open("/tmp/pointcloud_data.pcd", "wb") as f:
    f.write(objects[0].point_cloud)
pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd")
points = np.asarray(pcd.points)
Parameters:

camera_name (str) – The name of the camera

Returns:

The pointcloud objects with metadata

Return type:

List[viam.proto.common.PointCloudObject]

async get_properties(*, extra: Mapping[str, Any] | None = None, timeout: float | None = None) viam.services.vision.vision.Vision.Properties[source]

Get info about what vision methods the vision service provides. Currently returns boolean values that state whether the service implements the classification, detection, and/or 3D object segmentation methods.

::

# Grab the detector you configured on your machine my_detector = VisionClient.from_robot(robot, “my_detector”) properties = await my_detector.get_properties() properties.detections_supported # returns True properties.classifications_supported # returns False

Returns:

The properties of the vision service

Return type:

Properties

async do_command(command: Mapping[str, viam.utils.ValueTypes], *, timeout: float | None = None, **__) Mapping[str, viam.utils.ValueTypes][source]

Send/receive arbitrary commands.

motion = MotionClient.from_robot(robot, "builtin")

my_command = {
  "cmnd": "dosomething",
  "someparameter": 52
}

# Can be used with any resource, using the motion service as an example
await motion.do_command(command=my_command)
Parameters:

command (Dict[str, ValueTypes]) – The command to execute

Returns:

Result of the executed command

Return type:

Dict[str, ValueTypes]

classmethod from_robot(robot: viam.robot.client.RobotClient, name: str) typing_extensions.Self

Get the service named name from the provided robot.

async def connect() -> ViamClient:
    # Replace "<API-KEY>" (including brackets) with your API key and "<API-KEY-ID>" with your API key ID
    dial_options = DialOptions.with_api_key("<API-KEY>", "<API-KEY-ID>")
    return await ViamClient.create_from_dial_options(dial_options)

async def main():
    robot = await connect()

    # Can be used with any resource, using the motion service as an example
    motion = MotionClient.from_robot(robot=robot, name="builtin")

    robot.close()
Parameters:
  • robot (RobotClient) – The robot

  • name (str) – The name of the service

Returns:

The service, if it exists on the robot

Return type:

Self

classmethod get_resource_name(name: str) viam.proto.common.ResourceName

Get the ResourceName for this Resource with the given name

# Can be used with any resource, using an arm as an example
my_arm_name = my_arm.get_resource_name("my_arm")
Parameters:

name (str) – The name of the Resource

Returns:

The ResourceName of this Resource

Return type:

ResourceName

get_operation(kwargs: Mapping[str, Any]) viam.operations.Operation

Get the Operation associated with the currently running function.

When writing custom resources, you should get the Operation by calling this function and check to see if it’s cancelled. If the Operation is cancelled, then you can perform any necessary (terminating long running tasks, cleaning up connections, etc. ).

Parameters:

kwargs (Mapping[str, Any]) – The kwargs object containing the operation

Returns:

The operation associated with this function

Return type:

viam.operations.Operation

async close()

Safely shut down the resource and prevent further use.

Close must be idempotent. Later configuration may allow a resource to be “open” again. If a resource does not want or need a close function, it is assumed that the resource does not need to return errors when future non-Close methods are called.

await component.close()