RuntimeError: The size of tensor a (74) must match the size of tensor b (8) at non-singleton dimension 1

[acquywibucodoc]

hi i want to train Iudex, I composed up docker and ran python soulsai\distributed\client\main.py, but I encounter this problem:

PS E:\For the Dark Soul\SoulsAI> python soulsai\distributed\client\main.py
INFO:soulsai.distributed.client.dqn_client:Launching DQN client
INFO:soulsai.distributed.client.dqn_client:Press 'Enter' to end training
INFO:soulsai.distributed.client.connector:Waiting for model download...
INFO:soulsai.distributed.client.connector:Download complete, connector initialized
INFO:soulsai.distributed.client.dqn_client:Client node running
Traceback (most recent call last):
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\main.py", line 36, in
dqn_client(config)
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 52, in dqn_client
_dqn_client(config)
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 109, in _dqn_client
sample = con.transforms"obs"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\For the Dark Soul\SoulsAI\soulsai\core\transform.py", line 172, in forward
x[sample_key][..., idx] = (x[sample_key][..., idx] - mean[..., idx]) / std[..., idx]
~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (74) must match the size of tensor b (8) at non-singleton dimension 1
[W520 16:19:36.000000000 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

I tried to change many values in config.yaml but nothing seems to work:

Environment information

env:
name: SoulsGymIudex-v0
vectorize: False
kwargs:
game_speed: 1.
obs_shape: [74]
n_actions: 20
wrappers:
IudexObservationWrapper:
kwargs:

max_env_steps: null

Algorithm choice

algorithm: DQN

DQN parameters

dqn:
batch_size: 64
train_epochs: 25
update_samples: 25
fill_buffer: False
min_samples: null
max_model_delay: 10

agent:
type: DQNAgent
kwargs:
lr: 0.001
gamma: 0.99
multistep: 4
grad_clip: 1.
q_clip: 2
device: cuda

network:
type: DQN
kwargs:
input_dims: 74
output_dims: 20
layer_dims: 128

replay_buffer:
type: ReplayBuffer
kwargs:
max_size: 100_000

observation_transform:
type: Normalize
kwargs:
keys:
- obs
shapes:
- [8]

Can you look into this issue please? ty

[amacati]

Hi, your config seems off. The observation_transform seems off:

observation_transform:
type: Normalize
kwargs:
keys:

• obs
  shapes:
• [8]

Instead of having shapes [8], you should enter the real shape of your observation (74 in this case).

[acquywibucodoc]

I changed to 74 and it returns this error:

observation_transform:
type: Normalize
kwargs:
keys:
- obs
shapes:
- [74]

PS E:\For the Dark Soul\SoulsAI> python soulsai\distributed\client\main.py
INFO:soulsai.distributed.client.dqn_client:Launching DQN client
INFO:soulsai.distributed.client.dqn_client:Press 'Enter' to end training
INFO:soulsai.distributed.client.connector:Waiting for model download...
INFO:soulsai.distributed.client.connector:Download complete, connector initialized
INFO:soulsai.distributed.client.dqn_client:Client node running
Traceback (most recent call last):
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\main.py", line 36, in
dqn_client(config)
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 52, in dqn_client
_dqn_client(config)
File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 111, in _dqn_client
sample = con.transforms"action"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\For the Dark Soul\SoulsAI\soulsai\core\transform.py", line 411, in forward
tf_idx = torch.multinomial(self.params["probs"], x[key].shape[0], replacement=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)
[W520 18:29:31.000000000 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

[amacati]

Is the config above the full config? What action transforms do you use?

[acquywibucodoc]

Here's my full config, im sorry because im new to RL so i may not fully understand your config.

# Create config.yaml file to overwrite default settings

# General settings
loglevel: info

# Redis settings
redis_address: localhost

local:

# Environment information
env:
  name: SoulsGymIudex-v0
  vectorize: False
  kwargs:
    game_speed: 1.
  obs_shape: [74]
  n_actions: 20
  wrappers:
    IudexObservationWrapper:
      kwargs:

max_env_steps: null

# Algorithm choice
algorithm: DQN

# DQN parameters
dqn:
  batch_size: 64
  train_epochs: 25
  update_samples: 25
  fill_buffer: False
  min_samples: null
  max_model_delay: 10

  agent:
    type: DQNAgent
    kwargs:
      lr: 0.001
      gamma: 0.99
      multistep: 4
      grad_clip: 1.
      q_clip: 2
      device: cuda

  network:
    type: DQN
    kwargs:
      input_dims: 74
      output_dims: 20
      layer_dims: 128

  replay_buffer:
    type: ReplayBuffer
    kwargs:
      max_size: 100_000

  observation_transform:
    type: Normalize
    kwargs:
      keys:
        - obs
      shapes:
        - [74]

  value_transform:
    type: Chain
    kwargs:
      transforms:
        - type: Mask
          kwargs:
            key: __value__
            mask_key: [info, action_mask]
        - type: GreedyAction
          kwargs:
            value_key: __value__
            action_key: action

  action_transform:
    type: Choice
    kwargs:
      key: action
      probs: [0.9, 0.1]
      transforms:
        - type: Identity
        - type: ReplaceWithNoise
          kwargs:
            key: action
            noise:
              type: MaskedDiscreteNoise
              kwargs:
                mask_key: [info, action_mask]
                size_n: 20

# Training node settings
checkpoint:
  save: False
  epochs: 100  # Checkpoint every 100th iteration
  save_buffer: False
  load: False
  load_buffer: False
  load_config: False

# Telemetry node settings
telemetry:
  update_interval: 5
  moving_average: 20
  transforms:
    - type: MetricByKey
      kwargs:
        key: ep_reward
        name: rewards
    - type: MetricByKey
      kwargs:
        key: ep_steps
        name: steps
    - type: MetricByKey
      kwargs:
        key: total_steps
        name: n_env_steps
    - type: CompareValue
      kwargs:
        key: ep_reward
        name: wins
        value: 200
        op: gt  # Greater than
    - type: Timer
  log_keys:
    - rewards_av
    - steps_av
  callbacks:
    - type: SaveBest
      kwargs:
        key: rewards_av
        channel: save_best

# Monitoring
monitoring:
  file_storage:
    path: /home/SoulsAI/saves
    plot:
      xkey: n_env_steps
      ykeys:
        - rewards_av
        - steps_av
        - wins_av
        - time
  grafana: True
  prometheus: True
  # wandb:
  #   project: soulsai_iudex
  #   entity: amacati
  #   group: state_obs
  #   save_dir: /home/SoulsAI/saves

# Client settings
max_episodes: -1  # -1 for infinite runs
enable_interrupt: True
step_delay: 0

watchdog:
  enable: False
  minimum_samples: -1

[amacati]

Are you training on Windows with WSL2?

[amacati]

Can you check the values of self.params["probs"] in your code? Btw, SoulsAI is meant for distributed training on multiple worker machines. If you don't want to train with distributed worker nodes, it's way easier to use single-node frameworks like cleanrl.

[acquywibucodoc]

this is my first time using Docker, I try to follow your tutorial so I think I'm training on Windows with Docker Desktop. If possible, I want to only train on my local pc, so I might not need to train on multiple machines. btw, the self.params['probs']: Parameter containing is tensor([0., 0.])

INFO:soulsai.distributed.client.dqn_client:Launching DQN client
INFO:soulsai.distributed.client.dqn_client:Press 'Enter' to end training
INFO:soulsai.distributed.client.connector:Waiting for model download...
INFO:soulsai.distributed.client.connector:Download complete, connector initialized
INFO:soulsai.distributed.client.dqn_client:Client node running
self.params['probs']: Parameter containing:
tensor([0., 0.])  <----
Traceback (most recent call last):
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\main.py", line 36, in <module>
    dqn_client(config)
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 52, in dqn_client
    _dqn_client(config)
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 111, in _dqn_client
    sample = con.transforms["action"](sample)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl     
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\For the Dark Soul\SoulsAI\soulsai\core\transform.py", line 412, in forward
    tf_idx = torch.multinomial(self.params["probs"], x[key].shape[0], replacement=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)
[W520 21:33:58.000000000 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

[amacati]

Can you try to set your device to cpu and see if things change? This could be an instance of pytorch/pytorch#112340. See also https://docs.pytorch.org/docs/stable/notes/windows.html#cuda-ipc-operations

[acquywibucodoc]

I installed "conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 cpuonly -c pytorch" (CPU only) and it seems like the DQN requires cuda enabled. I'm using a nvidia gpu 4070ti super

PS E:\For the Dark Soul\SoulsAI> python soulsai\distributed\client\main.py                                                            
INFO:soulsai.distributed.client.dqn_client:Launching DQN client    
INFO:soulsai.distributed.client.dqn_client:Press 'Enter' to end training                                                              
Traceback (most recent call last):                                 
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\main.py", line 36, in <module>                                        
    dqn_client(config)                                             
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 52, in dqn_client
    _dqn_client(config)
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\dqn_client.py", line 80, in _dqn_client
    con = DQNConnector(config)
          ^^^^^^^^^^^^^^^^^^^^
  File "E:\For the Dark Soul\SoulsAI\soulsai\distributed\client\connector.py", line 76, in __init__
    self.agent = agent_cls(config.dqn.agent.type)(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\For the Dark Soul\SoulsAI\soulsai\core\agent.py", line 100, in __init__
    self.networks.add_module("dqn1", net_cls(network_type)(**network_kwargs).to(self.device))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1173, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 779, in _apply
    module._apply(fn)
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 804, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\nn\modules\module.py", line 1159, in convert
    return t.to(
           ^^^^^
  File "C:\Users\thand\anaconda3\envs\soulsAI_env\Lib\site-packages\torch\cuda\__init__.py", line 284, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")   
AssertionError: Torch not compiled with CUDA enabled

[acquywibucodoc]

hi i found the problem
I'm experiencing the same issue as described in this comment on PyTorch issue #112340. It may be causing the following error in my case.

[amacati]

You can set your desired training device in the config. There is a "device" keyword for the agent I believe

[Ekanara]

Yeah i face the same problem, though when i print self.params['probs'] in init, it show [0.9,0.1], but when i print self.params['probs'] in forward, it show [0.,0.], this is when i enable cuda only

class Choice(Transform):
    """Choice transformation class for selecting one of several transforms at random."""

    def __init__(self, key: NestedKey, transforms: list[Transform | dict], probs: list[float]):
        """Initialize the transformation.

        Args:
            key: Key(s) of the input tensor in the sample TensorDict.
            transforms: List of transformations.
            probs: List of probabilities for choosing each transformation.
        """
        super().__init__()
        self._key = key if isinstance(key, str) else tuple(key)

        for i, tf in enumerate(transforms):
            if isinstance(tf, dict):
                transforms[i] = transform_cls(tf["type"])(**(tf.get("kwargs") or {}))
        assert all(isinstance(x, Transform) for x in transforms), "All elements must be Transforms"

        self.params["transforms"] = nn.ModuleList(transforms)
        probs = torch.tensor(probs, dtype=torch.float32)
        assert torch.all(probs >= 0), "p must be non-negative"
        assert torch.isclose(probs.sum(), torch.tensor(1.0)), "p must sum to 1"
        assert len(probs) == len(transforms), "probs must have the same length as transforms"
        probs = probs / probs.sum()
        self.params["probs"] = nn.Parameter(probs, requires_grad=False)

    def forward(
        self, x: TensorDict, keys_mapping: dict[NestedKey, NestedKey] | None = None
    ) -> TensorDict:
        """Choose a transformation at random and apply it to the input tensor.

        Args:
            x: Sample TensorDict.
            keys_mapping: Optional dictionary that maps keys to new keys. This is useful when the
                transformation should be applied to a subset of the keys, or when the same
                parameters should be used for different keys, i.e. for 'obs' and 'next_obs'.

        Returns:
            Transformed TensorDict.
        """
        assert isinstance(x, TensorDict), f"Expected input to be a TensorDict, is {type(x)}"
        key = self._key if keys_mapping is None else keys_mapping[self._key]
        tf_idx = torch.multinomial(self.params["probs"], x[key].shape[0], replacement=True)
        x = torch.cat(
            [self.params["transforms"][j](x[[i]], keys_mapping) for i, j in enumerate(tf_idx)],
            dim=-1,
        )
        return x

[amacati]

Yes, this is an issue on PyTorch's side. Unfortunately there is not much I can do about it. The only way to work around it is to use the CPU as device on Windows. It's curious though because I think I previously had this issue only using WSL2, not on the Windows training clients.