Valve 6DoF estimation

[kluge7]

Goal: Replace Canny/Hough orientation with YOLO-predicted θ integrated into ROS pipeline and validated on test data.

Motivation:

• In the Valve Intervention task, we must turn two underwater valves to the Open/Shut positions the judge tells us.
• To do this autonomously (and earn bonus points), the robot must know each valve’s pose so it can grasp and turn correctly. Therefore, we need to estimate the valve’s pose.

Current solution:

• YOLOv8 (bbox) -> 2D location/extent (no orientation)
• Canny + Hough line detection -> in-plane handle orientation (θ)
• Depth/PCL (annulus + RANSAC plane) -> 3D point + surface normal (tilt)
• Fusion (θ + normal) -> full 6-DoF pose (position + handle orientation)

Problems with current solution:

• Canny + Hough -> fragile: fails under noise, poor lighting, turbidity; often returns NaN if no line is found.
• Depth-based angle estimation -> unreliable: depth noise and sparse/missing points distort plane fit and give unstable orientation.
• Manual angle detection -> not autonomous: requires human input, no scoring bonus, not scalable or robust.

Proposed solution**

• Extend YOLOv8 -> bbox + handle orientation (θ)
• Train with valve-angle annotations
• Keep depth/PCL for 3D position + tilt; fuse with YOLO angle for full pose

Suggested workflow

• Research and understand current solution for estimating in-plane handle orientation (θ)
• Ultralytics fork -> Evaluate the ultralytics fork that supports angle prediction. Verify if it can be adapted for valve orientation.
  • Need to figure out how to use it locally, just as it is now.
  • Need to figure out how to add a reghead
  • Write docs for how to add reghead
• Training setup -> Get familiar with Hugging Face, Roboflow, and Slurm.
• Annotation tool -> Build a simple tool to label images with valve angle (draw handle, calculate θ).
• Dataset annotation -> Use the tool to add angle labels to the existing dataset.
• Model training & testing -> Train YOLOv8 with bbox + θ, benchmark against current method.
• Integration -> Replace Canny/Hough with YOLO-predicted θ in ROS pipeline.

References

• Canny Edge Detection
• Hough Line Transform
• Roboflow
• Hugging Face
• Slurm example
• Ultralytics fork
• TAC ChallengeMission-Booklet-2024-rev.2
• RANSAC: https://www.geeksforgeeks.org/python/step-by-step-guide-to-using-ransac-in-opencv-using-python and https://acnice.medium.com/teach-me-the-ransac-algorithm-like-im-5-d3b1dd6d67a8
• Ultralytics: https://docs.ultralytics.com/quickstart/

[kluge7]

https://docs.ultralytics.com/tasks/obb/ - possible alternative?
@jenscaa

[kluge7]

Implement script for labeling for OBB and Keypoint.
@kluge7 cannyha higwh
@jenscaa will test and train up OBB and Keypoint

Then implement solution using the one that works best

[kluge7]

Planned setup:

• Assume we don't have stereo camera for depth estimation
• Use bounding boxes and pinhole camera model to estimate depth
• Use the grid to get normal vector
• Use canny edge detection + hough line transform to get angle (if it doesnt work on the image, instead rely on the angle from OBB)

@jenscaa

[kluge7]

Blocked until we confirm if the stereo camera has reliable depth estimation

[kluge7]

Make launch file and config file!

Here is the workflow:

• Make the node subscribe to Detection2D message, parse it and then publish landmark array with it. To simplify you can just retreieve the first Detection2D and publish that, but ideally it should publish both (however sometimes the bounding boxes are inside each other and are for the same valve, so you need handling for this so that only one of these are published as a landmark) so maximum you will publish 2 landmarks. If there are two valves detected, both of them should be published as a landmark, but if you get multiple bounding boxes for the same valve, only one of them should be published as a landmark. Also use intrinsics matrix and distortion to undistort image @kluge7 will send it.
• Retrieve depth from the Depth image, use the bounding box to get where in that image the valve is (to get accurate depth), and we want the depth of the ring on the valve. Here you also need to take into account transformations, since the Depth image is in a different frame than the color image - see here for the transformation (to make it simple, just hardcode the transformation in your code: URDF for D555 camera? realsenseai/librealsense#14577)

File layout idea.

• One ROS file which handles subscription and publishing (should have common types for parsing along files)
• One file which does the depth transformation stuff.
• One file for the normal vector estimation