1.Grasp Lable Format
1.1 Raw Lable Format （原始标签格式）
The raw label is composed of two parts, i.e. labels for all grasp candidates on each object and collision masks for each scene.
原始标签由两部分组成，每个物体的所有候选抓取标签and每个场景的collision masks。
（1）Labels on Objects
The raw label on each object is a list of numpy arrays.
每个物体的初始标签均为一个np.arry列表。
import numpy as np
import os
root_dir = '/media/robotics/data/datasets/GraspNet/'
data_dir1 = os.path.join(root_dir, 'grasp_label/000_labels.npz')
label = np.load(data_dir1) # GRASPNET_ROOT/grasp_label/000_labels.npz
print(label.files) # ['points', 'offsets', 'collision', 'scores']
print(label['points'].shape) # (3459, 3)
print(label['offsets'].shape) # (3459, 300, 12, 4, 3)
print(label['collision'].shape, # (3459, 300, 12, 4)
label['collision'].dtype) # dtype('bool')
print(label['scores'].shape, # (3459, 300, 12, 4)
label['scores'][0][0][0][0]) # -1.0
| name | 含义 | 翻译 |
|---|---|---|
| ‘points’ | records the grasp center point coordinates in model frame. |
记录抓取中心坐标。 |
| ‘offsets’ | records the in-plane rotation, depth and width of the gripper respectively in the last dimension. |
记录gripper的旋转、深度和宽度。 |
| ‘collision’ | records the bool mask for if the grasp pose collides with the model. |
记录布尔mask。 |
| ‘scores’ | records the minimum coefficient of friction between the gripper and object to achieve a stable grasp. |
记录gripper与物体之间的最小摩擦系数to实现稳定抓取。 |
Notes：
In the raw label, the lower score the grasp has, the better it is. However, -1.0 score means the grasp pose is totally not acceptable.
在原始标签中，grasp的分数越低越好，
但是，score = -1.0表示grasp pose完全不能被接受。
（2）Collision Masks on Each Scene
Collision mask on each scene is a list of numpy arrays.
每个场景的collision mask均为一个np.arry列表。
import numpy as np
data_dir2 = os.path.join(root_dir, 'collision_label/scene_0000/collision_labels.npz')
c_labels = np.load(data_dir2) # GRASPNET_ROOT/collision_label/scene_0000/collision_labels.npz
print(c_labels.files)
#['arr_0', 'arr_4', 'arr_5', 'arr_2', 'arr_3', 'arr_7', 'arr_1', 'arr_8', 'arr_6']
print(c_labels['arr_0'].shape, # (487, 300, 12, 4)
c_labels['arr_0'].dtype) # dtype('bool')
print(c_labels['arr_0'][10][20][3]) # array([ True, True, True, True])
‘arr_i’ is the collision mask for the i th object in the object_id_list.txt for each scene whose shape is (num_points, 300, 12, 4). num_points, 300, 12, 4 denote the number of points in the object, view id, in-plane rotation id and depth id respectively.
‘arr_i’是每个场景的object_id_list.txt中第i个物体的collision mask，其shape = (num points, 300, 12, 4)。
num_points, 300, 12, 4分别为点数、视角ID、内旋ID、深度ID。
更多细节请参考：graspnetAPI.GraspNet.loadGrasp()
1.2 API Loaded Labels
通过调用graspnetAPI.GraspNet.loadGrasp()，可以方便的get场景中的所以grasp labels、params、scores.
加载grasp labels有四种数据结构：Grasp, GraspGroup, RectGrasp and RectGraspGroup.
每个类的内部数据格式是np.arry数组，这比Python列表运算更高效。
它们的定义见 grasp.py
（1）Labels示例
Loading a GraspGroup instance.
加载一个GraspGroup实例
from graspnetAPI import GraspNet
import open3d as o3d
import cv2
# loading grasp for a scene.
graspnet_root = '/media/robotics/data/datasets/GraspNet' # ROOT PATH FOR GRASPNET
sceneId = 1
annId = 3
# 1.initialize a GraspNet instance
g = GraspNet(graspnet_root, camera='kinect', split='test_novel')
# load grasps of scene 1 with annotation id = 3, camera = kinect and fric_coef_thresh = 0.2
_6d_grasp = g.loadGrasp(sceneId = sceneId, annId = annId, format = '6d', camera = 'kinect', fric_coef_thresh = 0.2)
print('6d grasp:\n{}'.format(_6d_grasp))
Grasp Group, Number=90332:
Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.81045675 -0.57493848 0.11227506]
[ 0.49874267 -0.77775514 -0.38256073]
[ 0.30727136 -0.25405255 0.91708326]]
object id:66
Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.73440629 -0.67870212 0.0033038 ]
[ 0.64608938 -0.70059127 -0.3028869 ]
[ 0.20788456 -0.22030747 0.95302087]]
object id:66
Grasp: score:0.9000000357627869, width:0.08487851172685623, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.10412319 -0.13797761 0.38312319]
rotation:
[[ 0.03316294 0.78667933 -0.61647028]
[-0.47164679 0.55612749 0.68430358]
[ 0.88116372 0.26806271 0.38947761]]
object id:66
通过index or slice访问元素
# index（索引）
print('_6d_grasp[0](grasp):\n{}'.format(_6d_grasp[0]))
# slice（切片）
print('_6d_grasp[0:2]:\n{}'.format(_6d_grasp[0:2]))
print('_6d_grasp[[0,1]]:\n{}'.format(_6d_grasp[[0,1]]))
_6d_grasp[0] (grasp):
Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.81045675 -0.57493848 0.11227506]
[ 0.49874267 -0.77775514 -0.38256073]
[ 0.30727136 -0.25405255 0.91708326]]
object id:66
_6d_grasp[0:2]:
Grasp Group, Number=2:
Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.81045675 -0.57493848 0.11227506]
[ 0.49874267 -0.77775514 -0.38256073]
[ 0.30727136 -0.25405255 0.91708326]]
object id:66
Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.73440629 -0.67870212 0.0033038 ]
[ 0.64608938 -0.70059127 -0.3028869 ]
[ 0.20788456 -0.22030747 0.95302087]]
object id:66
_6d_grasp[[0,1]]:
Grasp Group, Number=2:
Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.81045675 -0.57493848 0.11227506]
[ 0.49874267 -0.77775514 -0.38256073]
[ 0.30727136 -0.25405255 0.91708326]]
object id:66
Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084 0.39480919]
rotation:
[[-0.73440629 -0.67870212 0.0033038 ]
[ 0.64608938 -0.70059127 -0.3028869 ]
[ 0.20788456 -0.22030747 0.95302087]]
object id:66
GraspGroup的每个元素都是一个Grasp实例，
Grasp的属性可以通过提供的方法访问
# grasp is a Grasp instance defined in grasp.py
# access and set properties.
grasp = _6d_grasp[0]
print('grasp.translation={}'.format(grasp.translation))
grasp.translation = np.array([1.0, 2.0, 3.0])
print('After modification, grasp.translation={}'.format(grasp.translation))
print('grasp.rotation_matrix={}'.format(grasp.rotation_matrix))
grasp.rotation_matrix = np.eye(3).reshape((9))
print('After modification, grasp.rotation_matrix={}'.format(grasp.rotation_matrix))
print('grasp.width={}, height:{}, depth:{}, score:{}'.format(grasp.width, grasp.height, grasp.depth, grasp.score))
grasp.translation=[-0.09166837 -0.16910084 0.39480919]
After modification, grasp.translation=[1. 2. 3.]
grasp.rotation_matrix=[[-0.81045675 -0.57493848 0.11227506]
[ 0.49874267 -0.77775514 -0.38256073]
[ 0.30727136 -0.25405255 0.91708326]]
After modification, grasp.rotation_matrix=[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]
grasp.width=0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, score:0.9000000357627869
RectGrasp是矩形抓取的类，
格式与Grasp不同，但提供的API类似。
# load rectangle grasps of scene 1 with annotation id = 3, camera = realsense and fric_coef_thresh = 0.2
rect_grasp_group = g.loadGrasp(sceneId = sceneId, annId = annId, format = 'rect', camera = 'realsense', fric_coef_thresh = 0.2)
print('rectangle grasp group:\n{}'.format(rect_grasp_group))
# rect_grasp is an RectGraspGroup instance defined in grasp.py
print('rect_grasp_group:\n{}'.format(rect_grasp_group))
# index
rect_grasp = rect_grasp_group[0]
print('rect_grasp_group[0](rect_grasp):\n{}'.format(rect_grasp))
# slice
print('rect_grasp_group[0:2]:\n{}'.format(rect_grasp_group[0:2]))
print('rect_grasp_group[[0,1]]:\n{}'.format(rect_grasp_group[[0,1]]))
# properties of rect_grasp
print('rect_grasp.center_point:{}, open_point:{}, height:{}, score:{}'.format(rect_grasp.center_point, rect_grasp.open_point, rect_grasp.height, rect_grasp.score))
rectangle grasp group:
----------
Rectangle Grasp Group, Number=29036:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.6000000238418579, height:52.63461685180664, open point:(649.1359, -68.900116), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.30000001192092896, height:52.629859924316406, open point:(651.5441, -67.304474), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.800000011920929, height:52.84688186645508, open point:(625.3405, -55.22821), center point:(560.52704, -127.61005), object id:66
......
Rectangle Grasp: score:0.30000001192092896, height:40.82650375366211, open point:(613.7542, 459.35962), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.40000003576278687, height:40.82828140258789, open point:(749.2127, 549.03204), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.6000000238418579, height:39.991939544677734, open point:(584.6618, 410.68518), center point:(658.4677, 494.0827), object id:14
Rectangle Grasp: score:0.5, height:39.60657501220703, open point:(596.4554, 400.3207), center point:(670.0193, 483.46442), object id:14
Rectangle Grasp: score:0.40000003576278687, height:39.93021774291992, open point:(602.8676, 408.1247), center point:(673.44824, 487.8963), object id:14
----------
rect_grasp_group:
----------
Rectangle Grasp Group, Number=29036:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.6000000238418579, height:52.63461685180664, open point:(649.1359, -68.900116), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.30000001192092896, height:52.629859924316406, open point:(651.5441, -67.304474), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.800000011920929, height:52.84688186645508, open point:(625.3405, -55.22821), center point:(560.52704, -127.61005), object id:66
......
Rectangle Grasp: score:0.30000001192092896, height:40.82650375366211, open point:(613.7542, 459.35962), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.40000003576278687, height:40.82828140258789, open point:(749.2127, 549.03204), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.6000000238418579, height:39.991939544677734, open point:(584.6618, 410.68518), center point:(658.4677, 494.0827), object id:14
Rectangle Grasp: score:0.5, height:39.60657501220703, open point:(596.4554, 400.3207), center point:(670.0193, 483.46442), object id:14
Rectangle Grasp: score:0.40000003576278687, height:39.93021774291992, open point:(602.8676, 408.1247), center point:(673.44824, 487.8963), object id:14
----------
rect_grasp_group[0](rect_grasp):
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
rect_grasp_group[0:2]:
----------
Rectangle Grasp Group, Number=2:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
----------
rect_grasp_group[[0,1]]:
----------
Rectangle Grasp Group, Number=2:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
----------
rect_grasp.center_point:(560.52704, -127.61005), open_point:(677.3685, -87.56467), height:52.59913635253906, score:0.40000003576278687
（2）6D Grasp
17 个浮点数用于定义通用的 6d grasp，width、height、 depth、 score and 附加object id也是定义的一部分。
Notes：
抓取的分数越高越好，这与原始标签不同。
实际上，score = 1.1 - raw_score（摩擦系数）
class Grasp():
def __init__(self, *args):
'''
**Input:**
- args can be a numpy array or tuple of the score, width, height, depth, rotation_matrix, translation, object_id
- the format of numpy array is [score, width, height, depth, rotation_matrix(9), translation(3), object_id]
- the length of the numpy array is 17.
'''
if len(args) == 0:
self.grasp_array = np.array([0, 0.02, 0.02, 0.02, 1, 0, 0, 0, 1 ,0 , 0, 0, 1, 0, 0, 0, -1], dtype = np.float64)
elif len(args) == 1:
if type(args[0]) == np.ndarray:
self.grasp_array = copy.deepcopy(args[0])
else:
raise TypeError('if only one arg is given, it must be np.ndarray.')
elif len(args) == 7:
score, width, height, depth, rotation_matrix, translation, object_id = args
self.grasp_array = np.concatenate([np.array((score, width, height, depth)),rotation_matrix.reshape(-1), translation, np.array((object_id)).reshape(-1)]).astype(np.float64)
else:
raise ValueError('only 1 or 7 arguments are accepted')
（3）6D Grasp Group
通常，一个场景中有很多grasps，这些抓点组成一个类GraspGroup。
相比Grasp，GraspGroup包含一个二维 numpy 数组，额外的维度是每个grasp的索引。
class GraspGroup():
def __init__(self, *args):
'''
**Input:**
- args can be (1) nothing (2) numpy array of grasp group array (3) str of the npy file.
'''
if len(args) == 0:
self.grasp_group_array = np.zeros((0, GRASP_ARRAY_LEN), dtype=np.float64)
elif len(args) == 1:
if isinstance(args[0], np.ndarray):
self.grasp_group_array = args[0]
elif isinstance(args[0], str):
self.grasp_group_array = np.load(args[0])
else:
raise ValueError('args must be nothing, numpy array or string.')
else:
raise ValueError('args must be nothing, numpy array or string.')
实现了对列表的常见操作，例如s indexing, slicing and sorting。
此外，一个重要的功能是用户可以将GraspGroup转储到一个 numpy 文件中，并通过调用GraspGroup.save_npy()和GraspGroup.from_npy()将其加载到另一个程序中。
（4）Rectangle Grasp
7个浮点数用于定义通用的矩形抓取，即the center point, the open point, height, score and the
attached object id.。每个参数的详细定义如上图所示，中心点和开放点的坐标在像素框中。
class RectGrasp():
def __init__(self, *args):
'''
**Input:**
- args can be a numpy array or tuple of the center_x, center_y, open_x, open_y, height, score, object_id
- the format of numpy array is [center_x, center_y, open_x, open_y, height, score, object_id]
- the length of the numpy array is 7.
'''
if len(args) == 1:
if type(args[0]) == np.ndarray:
self.rect_grasp_array = copy.deepcopy(args[0])
else:
raise TypeError('if only one arg is given, it must be np.ndarray.')
elif len(args) == RECT_GRASP_ARRAY_LEN:
self.rect_grasp_array = np.array(args).astype(np.float64)
else:
raise ValueError('only one or six arguments are accepted')
（5）Rectangle Grasp Group
RectGraspGroup的格式类似于RectGrasp和GraspGroup
class RectGraspGroup():
def __init__(self, *args):
'''
**Input:**
- args can be (1) nothing (2) numpy array of rect_grasp_group_array (3) str of the numpy file.
'''
if len(args) == 0:
self.rect_grasp_group_array = np.zeros((0, RECT_GRASP_ARRAY_LEN), dtype=np.float64)
elif len(args) == 1:
if isinstance(args[0], np.ndarray):
self.rect_grasp_group_array = args[0]
elif isinstance(args[0], str):
self.rect_grasp_group_array = np.load(args[0])
else:
raise ValueError('args must be nothing, numpy array or string.')
else:
raise ValueError('args must be nothing, numpy array or string.')
1.3 Grasp and GraspGroup 转换
可以转换 Grasp 或 GraspGroup 得到 4x4 矩阵。
from graspnetAPI import *
# transform grasp
g = Grasp() # simple Grasp
frame = o3d.geometry.TriangleMesh.create_coordinate_frame(0.1)
# Grasp before transformation
o3d.visualization.draw_geometries([g.to_open3d_geometry(), frame])
g.translation = np.array((0,0,0.01))
# setup a transformation matrix
T = np.eye(4)
T[:3,3] = np.array((0.01, 0.02, 0.03))
T[:3,:3] = np.array([[0,0,1.0],[1,0,0],[0,1,0]])
g.transform(T)
# Grasp after transformation
o3d.visualization.draw_geometries([g.to_open3d_geometry(), frame])
g1 = Grasp()
gg = GraspGroup()
gg.add(g)
gg.add(g1)
# GraspGroup before transformation
o3d.visualization.draw_geometries([*gg.to_open3d_geometry_list(), frame])
gg.transform(T)
# GraspGroup after transformation
o3d.visualization.draw_geometries([*gg.to_open3d_geometry_list(), frame])