RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x8 and 10x256). Proprio Input : 10 vs Actual size 8

[mizuh0n]

Hi, Thanks for your great work and now I got a self trained checkpoint but have an error.

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x8 and 10x256)

  File "/work/modules/otter/script/eval_libero.py", line 247, in <module>
    eval_libero()                                                                                                                                                                                                                             File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/draccus/argparsing.py", line 228, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/work/modules/otter/script/eval_libero.py", line 182, in eval_libero
    action = model(                                                                                                                                                                                                                           File "/work/modules/python/otter/policy/otter_interface.py", line 298, in __call__
    action = self.model.forward_inference(                                                                                                                                                                                                    File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/work/modules/python/otter/policy/otter.py", line 319, in forward_inference
    transformer_input = self.forward_encoder(images, text, proprio, text_mask)
  File "/work/modules/python/otter/policy/otter.py", line 258, in forward_encoder
    proprio_token = self.proprio_encoder(proprio)  # (B*T, proprio_dim)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/work/modules/python/otter/policy/models.py", line 53, in forward
    return self.mlp(x)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward
    input = module(input)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/work/x86_64_24.05-py3/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
  RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x8 and 10x256)

This is because the shape of proprio input to model is actually (8,) (=3+3+2)

otter/script/eval_libero.py

Lines 176 to 178 in 4be49c4

	proprio=np.concatenate(
	(obs["robot0_eef_pos"], quat2axisangle(obs["robot0_eef_quat"]), obs["robot0_gripper_qpos"])
	)

but the model has the proprio size 10. How to solve this error?

Loading model from checkpoint: data/weights/checkpoint_75000.pt
======================================================================
                             OTTER Model
======================================================================

                          Model Architecture
----------------------------------------------------------------------
CLIP Model     : ViT-L/14
Cameras        : 2 (image_primary, image_wrist)
Seq Length     : 12
Action Horizon : 12

                              Dimensions
----------------------------------------------------------------------
Proprio Input                : 10
Action Dim                   : 10
Num Readouts                 : 4
First K Tokens               : 15
Vision Pool Out (per camera) : 256
Text Pool Out                : 128
Proprio Out                  : 64
FT Dim                       : 704
Transformer In               : 768

                              Parameters
----------------------------------------------------------------------
Total                           : 458.67M
CLIP                            : 427.62M
Policy                          : 28.37M
Trainable (incl. modality enc.) : 0

======================================================================

[FangchenLiu]

Hi,

This example code snippet is initially adapted from OpenVLA's evaluation in LIBERO, which uses the position, axisangle, and gripper states together to form an 8-dimensional input (3 + 3 + 2). Our default configuration (used in real-world experiments) represents rotation with a Rot6D along with position and gripper open/close, resulting in a 10-dimensional input.

You can either update the configuration to use 8 dimensions or modify the LIBERO evaluation code to support 10. We'll soon fix this part and release our newest sim models. Please stay tuned :)

[mizuh0n]

Thank you for your reply and sorry for my noob question. Yes, I tried LIBERO evaluation on (Non LIBERO dataset) trained weight.
If you provide LIBERO eval/train code (and weight?), it's Great!.

What kind robot your default configuration assumes? Models provided in mujoco_menagerie can be used?
update: I checked the OTTER dataset and looks Franka Emika is used.

[mizuh0n]

@FangchenLiu Hi!
Is the output actions from the model is EEF position? velocity?

I got actions like this and found that is very small value series....

# Do easy inference by using OTTER dataset images and proprio
import otter.dataset.utils as otter_utils
import otter.data.libero.libero_utils as libero_utils

images = {k: libero_utils.resize_image(v, resize_size) for k, v in frames[0].items()}
task_description = 'move on the red cube'
tmp = proprios[0]
rot6d = otter_utils.euler_to_rot_6d(tmp[3:6][None, ...])
proprio = np.concatenate([tmp[:3], rot6d[0], tmp[-1:]])


pred_action = otter_model(
    images=images,
    text=task_description,
    proprio=proprio
)

pred_action.shape # (12, 10)

# did something
np.array(pred_action7).mean(0) # array([0.009, 0.007, 0.037, -0.021, -0.004, -0.029, -0.002])