The model loaded with the (ZijianZhang/OpenVLA-7B-GRAPE-Simpler) failed in SimplerEnv environment

[V-Enzo]

Hi,

Thank you for open-sourcing your work, but I met some problems in reproducing the performance on SimplerEnv.

I followed the requirements and settings described in the README to set up the environment.

I try to run the first experiment in Simpler-env.

python simpler_env/main_inference.py \                                                                                                            
  --policy-model openvla \
  --ckpt-path "/home/charles/Desktop/GRAPE/OpenVLA-7B-GRAPE-Simpler" \
  --robot widowx \
  --policy-setup widowx_bridge \
  --control-freq 5 \
  --sim-freq 500 \
  --max-episode-steps 100 \
  --env-name PutCarrotOnPlateInScene-v0 \
  --scene-name bridge_table_1_v1 \
  --rgb-overlay-path ./ManiSkill2_real2sim/data/real_inpainting/bridge_real_eval_1.png \
  --robot-init-x 0.147 0.147 1 \
  --robot-init-y 0.028 0.028 1 \
  --obj-variation-mode episode \
  --obj-episode-range 0 10 \
  --robot-init-rot-quat-center 0 0 0 1 \
  --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1

But all 50 episodes fail at this task, with the gripper moving upwards and disappearing in the scene.

like generating videos as

failure_obj_episode_1_moved_correct_obj_False_moved_wrong_obj_False_is_src_obj_grasped_False_consecutive_grasp_False_src_on_target_False.mp4

I have already modified the unnorm_key as "bridge_orig" in the openvla_model.py at line 26 as

        if policy_setup == "widowx_bridge":
            unnorm_key = "bridge_orig" if unnorm_key is None else unnorm_key

And here is the info about my system.

Ubuntu 22.04.5 LTS
GPU: NVIDIA GeForce RTX 4090
Driver Version: 550.163.01 
torch: 2.2.0 + CUDA: 12.1

I am new to VLA, and any help is appreciated!

Thank you very much!