control node warnings and errors after launching vision nodes

[vamsikalagaturu]

Description

When kinova_vision is launched after launching the kortex_bringup, the controller_manager starts throwing warings

[ros2_control_node-2] [WARN] [1759133281.192190739] [controller_manager]: Overrun might occur, Total time : 28555.258 us (Expected < 10000.000 us) --> Read time : 11.913 us, Update time : 28057.236 us (Switch time : 28044.852 us (Switch chained mode time : 3.733 us, perform mode change time : 23854.631 us, Activation time : 4177.968 us, Deactivation time : 2.031 us)), Write time : 486.109 us

[ros2_control_node-2] [WARN] [1759133281.192232004] [controller_manager]: Overrun detected! The controller manager missed its desired rate of 100 Hz. The loop took 28.668771 ms (missed cycles : 3).

and, when pointcloud is subscribed (/camera/depth/color/points), KortexMultiInterfaceHardware throws

Runtime error: timeout detected: BaseCyclicClient::Refresh

which then leads to unloading/deactivating of controllers, and sometimes even shutting down the node

[ros2_control_node-2] [ERROR] [1759133353.353892787] [controller_manager]: Exception of type : St13runtime_error thrown during write of the component 'kinova_KortexMultiInterfaceHardware': timeout detected: BaseCyclicClient::RefreshFeedback
[ros2_control_node-2]
[ros2_control_node-2] [ERROR] [1759133353.353986042] [controller_manager]: Deactivating following hardware components as their write cycle resulted in an error: [ kinova_KortexMultiInterfaceHardware ]
[ros2_control_node-2] [ERROR] [1759133353.353994292] [controller_manager]: Deactivating following controllers as their hardware components write cycle resulted in an error: [ robotiq_gripper_controller fault_controller joint_state_broadcaster joint_trajectory_controller ]
[ros2_control_node-2] [ERROR] [1759133353.354054342] [controller_manager]: Not acceptable command interfaces combination:
[ros2_control_node-2] Start interfaces:
[ros2_control_node-2] [
[ros2_control_node-2] ]
[ros2_control_node-2] Stop interfaces:
[ros2_control_node-2] [
[ros2_control_node-2]   kinova_robotiq_85_left_knuckle_joint/position
[ros2_control_node-2]   reset_fault/command
[ros2_control_node-2]   reset_fault/async_success
[ros2_control_node-2]   kinova_joint_1/position
[ros2_control_node-2]   kinova_joint_2/position
[ros2_control_node-2]   kinova_joint_3/position
[ros2_control_node-2]   kinova_joint_4/position
[ros2_control_node-2]   kinova_joint_5/position
[ros2_control_node-2]   kinova_joint_6/position
[ros2_control_node-2]   kinova_joint_7/position
[ros2_control_node-2] ]
[ros2_control_node-2] Not available:
[ros2_control_node-2] [
[ros2_control_node-2]  kinova_robotiq_85_left_knuckle_joint/position
[ros2_control_node-2]  reset_fault/command
[ros2_control_node-2]  reset_fault/async_success
[ros2_control_node-2]  kinova_joint_1/position
[ros2_control_node-2]  kinova_joint_2/position
[ros2_control_node-2]  kinova_joint_3/position
[ros2_control_node-2]  kinova_joint_4/position
[ros2_control_node-2]  kinova_joint_5/position
[ros2_control_node-2]  kinova_joint_6/position
[ros2_control_node-2]  kinova_joint_7/position
[ros2_control_node-2] ]
[ros2_control_node-2]
[ros2_control_node-2] [ERROR] [1759133353.354065561] [controller_manager]: Error while attempting mode switch when deactivating controllers in write cycle!

At the same time, kinova_vision also struggles to start the stream for most of the time and eventually connects which then makes the ros2_control node to throw errors

[kinova_vision_node-1] [ERROR] [1759133465.820580987] [camera.kinova_vision_depth]: [depth]: Failed to start stream
[kinova_vision_node-1] [INFO] [1759133465.821783711] [camera.kinova_vision_depth]: [depth]: Trying to connect... (attempt #1)
[kinova_vision_node-2] [ERROR] [1759133465.870369484] [camera.kinova_vision_color]: [color]: Failed to start stream
[kinova_vision_node-2] [INFO] [1759133465.871341497] [camera.kinova_vision_color]: [color]: Trying to connect... (attempt #1)
[kinova_vision_node-1] [WARN] [1759133468.822601641] [camera.kinova_vision_depth]: [depth]: Stream is PAUSED
[kinova_vision_node-2] [WARN] [1759133468.876550022] [camera.kinova_vision_color]: [color]: Stream is PAUSED

Steps to reproduce

versions

• OS: Ubuntu 24.04
• ROS 2: Jazzy
• ROS 2 middleware: cyclone_dds
• ros2_kortex: commit
• kortex_vision: commit

commands

1. Launch the robot bringup

   ros2 launch kortex_bringup gen3.launch.py robot_ip:=x.x.x.x gripper:=robotiq_2f_85

2. Launch the vision nodes

   ros2 launch kinova_vision kinova_vision.launch.py device:=x.x.x.x  depth_registration:=true

3. Subscribe to Pointcloud (/camera/depth/color/points) in rviz or echo.

• I have tried with different camera resolutions from high to low, which had no effect.
• The errors happen only when pointcloud is subscribed. Otherwise, only warnings occur.
• Attached are the logs for both cases.

ros2_kortex_control_node_issue.tar.gz

Next steps

• I will try zenoh as middleware and update the issue with its results

[vamsikalagaturu]

Update

Using Zenoh as middleware

• zenoh_rmw

Resulting behavior

• The control node no longer throws any errors after the pointcloud is subscribed. The Overrun detected warnings persists though at 1KHz but not at lower frequencies. zenoh_pc_sub_logs.tar.gz

Note

Switching to Zenoh will be a temporary fix till the underlying issue is fixed