Some observations during test

[ge79puv]

Hi @xukechun,

thanks for your great job! I have tested your repo successfully, but I also notice some issues during my test.

First of all, sometimes it seems that the robot can directly grasp the target object theoretically. However, the robot always tends to grasp other objects for several times, and even some objects are far away from the target object. I guess it's because the output of logits is not accurate enough. In my opinion, the selectivity of the grasp object is not very obvious.

Moreover, when the robot tries to grasp an object, it usually touches the object and then the object slips away, which results in the failure of grasp. I think if we try to use better pretrained grasp model, it can be improved.

In addition, for the real world experiments, should we also need to obtain the segmentation masks? Because for the training and test in the Pybullet simulator, we need to get all the "color_image, depth_image, mask_image". Should we add a segmentation model for the real world test? And in the real world, could we directly use the "trained_model.pth" from https://drive.google.com/drive/folders/1LCuoXX92X8L9wqJTbVqvskjRhTJrDDay?usp=sharing? Which part of your code should be changed for the real world test?

I am not sure if my observation and analysis is correct. It would be great if you can give me some advices :)

Best regards

[ge79puv]

I also notice that you use the true object poses of "env.get_true_object_poses()" as input of graspnet.grasp_detection(). Which algorithm do you use to get the object poses in the real world test?

[xukechun]

First of all, sometimes it seems that the robot can directly grasp the target object theoretically. However, the robot always tends to grasp other objects for several times, and even some objects are far away from the target object. I guess it's because the output of logits is not accurate enough. In my opinion, the selectivity of the grasp object is not very obvious.

Yes, sometimes the ambiguity of language instruction may mislead the grasp selection, leading to the grasp of other objects even far from the target. This can be improved by introducing better observation and policy representation, and we have worked on such topics recently. We will release our new work soon, please stay tuned!

Moreover, when the robot tries to grasp an object, it usually touches the object and then the object slips away, which results in the failure of grasp. I think if we try to use better pretrained grasp model, it can be improved.

That's true, better grasp models, such as AnyGrasp, may improve the performance. And a new grasp model can easily plug into our method.

In addition, for the real world experiments, should we also need to obtain the segmentation masks? Because for the training and test in the Pybullet simulator, we need to get all the "color_image, depth_image, mask_image". Should we add a segmentation model for the real world test? And in the real world, could we directly use the "trained_model.pth" from https://drive.google.com/drive/folders/1LCuoXX92X8L9wqJTbVqvskjRhTJrDDay?usp=sharing? Which part of your code should be changed for the real world test?

I also notice that you use the true object poses of "env.get_true_object_poses()" as input of graspnet.grasp_detection(). Which algorithm do you use to get the object poses in the real world test?

In the real-world experiment, we use Openset-RCNN for object detection, and we directly use all the bounding boxes to filter grasp poses out of the objects, without the need of object pose estimation. Upon receiving the RGB-D images, we crop the images within the workspace, and get the detection results from Openset-RCNN. With the detection results, we run our algorithm with the pretrained model to generate a grasp pose to execute. We use ROS to communicate among different modules. I will add some instructions for real-world experiments in README later.