viam.services.vision
Module Contents
Classes
str(object='') -> str |
|
Connect to the Vision service, which allows you to access various computer vision algorithms |
- class viam.services.vision.VisModelType[source]
str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.
- DETECTOR_TF_LITE = tflite_detector
- DETECTOR_TENSORFLOW = tf_detector
- DETECTOR_COLOR = color_detector
- CLASSIFIER_TFLITE = tflite_classifier
- CLASSIFIER_TENSORFLOW = tf_classifier
- DETECTOR_SEGMENTER = detector_segmenter
- RADIUS_CLUSTERING_SEGMENTER = radius_clustering_segmenter
- class viam.services.vision.VisModelConfig[source]
- name :str
- type :VisModelType
- parameters :Mapping[str, Any]
- class viam.services.vision.VisionServiceClient(name: str, channel: grpclib.client.Channel)[source]
Connect to the Vision service, which allows you to access various computer vision algorithms (like detection, segmentation, tracking, etc) that usually only require a camera or image input.
- SERVICE_TYPE = vision
- async get_detector_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str] [source]
Get the list of detectors currently registered in the service.
- Returns
The detector names
- Return type
List[str]
- async add_detector(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Add a new detector to the service. Returns nothing if successful, and an error if not. Registers a new detector just as if you had put it in the original “register_models” field in the robot config. Available types and their parameters can be found in the vision service documentation.
- Parameters
config (VisModelConfig) – The configuration of the detector to add.
- async remove_detector(detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Remove the detector with the given name from the service. Returns nothing if successful.
- Parameters
detector_name (str) – The name of the detector to remove
- async get_detections_from_camera(camera_name: str, detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Detection] [source]
Get a list of detections in the next image given a camera and a detector
- Parameters
camera_name (str) – The name of the camera to use for detection
detector_name (str) – The name of the detector to use for detection
- Returns
A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.
- Return type
List[Detection]
- async get_detections(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], detector_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Detection] [source]
Get a list of detections in the given image using the specified detector
- Parameters
image (Image) – The image to get detections from
detector_name (str) – The name of the detector to use for detection
- Returns
A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.
- Return type
List[Detection]
- async get_classifier_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str] [source]
Get the list of classifiers currently registered to the service
- Returns
The list of classifier names
- Return type
List[str]
- async add_classifier(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Add a classifier to the service.
- Parameters
config (VisModelConfig) – The configuration of the classifier
- async remove_classifier(classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Remove the classifier with the given name from the service. Returns nothing if successful.
- Parameters
classifier_name (str) – The name of the classifier to remove
- async get_classifications_from_camera(camera_name: str, classifier_name: str, count: int, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Classification] [source]
Get a list of classifications in the next image given a camera and a classifier
- Parameters
camera_name (str) – The name of the camera to use for detection
classifier_name (str) – The name of the classifier to use for classification
count (int) – The number of classifications desired
- Returns
The list of Classifications
- Return type
List[Classification]
- async get_classifications(image: Union[viam.media.viam_rgba_plugin.Image.Image, viam.media.video.RawImage], classifier_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.service.vision.Classification] [source]
Get a list of detections in the given image using the specified detector
- Parameters
image (Image) – The image to get detections from
classifier_name (str) – The name of the detector to use for detection
- Returns
The list of Classifications
- Return type
List[Classification]
- async get_segmenter_names(*, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[str] [source]
Get the list of segmenters currently registered in the service.
- Returns
The segmenter names
- Return type
List[str]
- async add_segmenter(config: VisModelConfig, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Add a segmenter to the service
- Parameters
config (VisModelConfig) – The configuration of the segmenter
- async remove_segmenter(segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None)[source]
Remove the segmenter with the given name from the service. Returns nothing if successful.
- Parameters
segmenter_name (str) – The name of the segmenter to remove
- async get_model_parameters_schema(model_type: VisModelType, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) Mapping[str, Union[str, int, float, bool, Sequence, Mapping]] [source]
Get the parameters needed to add a model to the vision registry.
- Parameters
model_type (VisModelType) – The name of model
- Returns
A dictionary representing the parameters as JSONSchema
- Return type
Mapping[str, str | int | float | bool | Sequence | Mapping]
- async get_object_point_clouds(camera_name: str, segmenter_name: str, *, extra: Optional[Mapping[str, Any]] = None, timeout: Optional[float] = None) List[viam.proto.common.PointCloudObject] [source]
Returns a list of the 3D point cloud objects and associated metadata in the latest picture obtained from the specified 3D camera (using the specified segmenter).
To deserialize the returned information into a numpy array, use the Open3D library.
import numpy as np import open3d as o3d object_point_clouds = await vision.get_object_point_clouds(camera_name, segmenter_name) # write the first object point cloud into a temporary file with open("/tmp/pointcloud_data.pcd", "wb") as f: f.write(object_point_clouds[0].point_cloud) pcd = o3d.io.read_point_cloud("/tmp/pointcloud_data.pcd") points = np.asarray(pcd.points)
- Parameters
camera_name (str) – The name of the camera
segmenter_name (str) – The name of the segmenter
- Returns
The pointcloud objects with metadata
- Return type
List[PointCloudObject]