update the generation code compatible with ai2thor 2.4.0 (latest)

[jiasenlu]

No description provided.

[MohitShridhar]

Thanks @jiasenlu! I'll look into this soon.

There are couple of issues with upgrading to 2.4.0:

• A few receptacles were updated/added on the Unity-side. So some of the referring expressions collected with 2.1.0 might not be valid with 2.4.0.
• API changes. This should be easy to fix; involves some grunt work. But ALFRED relies on some special functions in 2.1.0 for restoring objects with a list of locations and object states.

[jiasenlu]

Thanks @jiasenlu! I'll look into this soon.

There are couple of issues with upgrading to 2.4.0:

• A few receptacles were updated/added on the Unity-side. So some of the referring expressions collected with 2.1.0 might not be valid with 2.4.0.
• API changes. This should be easy to fix; involves some grunt work. But ALFRED relies on some special functions in 2.1.0 for restoring objects with a list of locations and object states.

yes, I just fixed some API changes of Thor and it can generate scene for all goals. could you verify that the new code work properly? Thanks!

[jiasenlu]

@MohitShridhar one quick question, when doing a reply check, what is the success Rate? I do see some error that some object is blocking the way I wonder is this common?

---

Success Rate: 84.50/122.50 = 0.690
Non-deterministic failure: 0/38 = 0.000
Failures by crash: 36/38 = 0.947
Failures by unsatisfied: 2/38 = 0.053
Failures by json decode error: 0/38 = 0.000

This is the log for failed trajectory.

check 1/2 for file 'data/new_trajs/pick_and_place_with_movable_recep-Spoon-Cup-SideTable-28/trial_T20200922_223616_980108/traj_data.json'
Resetting ThorEnv
Traceback (most recent call last):
File "scripts/replay_checks.py", line 93, in replay_check
steps_taken = replay_json(env, json_file)
File "/home/jiasenl/code/alfred_new/gen/utils/replay_json.py", line 48, in replay_json
raise RuntimeError(err)
RuntimeError: DiningTable_9a2975d0 is blocking Agent 0 from moving 0
%%%%%%%%%%
errors (36):
(0.17) (6) Chair_1dc8a6a7 is blocking Agent 0 from moving 0
(0.08) (3) Chair_74a002dc is blocking Agent 0 from moving 0
(0.08) (3) DiningTable_098d08fe is blocking Agent 0 from moving 0
(0.08) (3) Sofa_8e2f971e is blocking Agent 0 from moving 0
(0.08) (3) Chair_eb06af7d is blocking Agent 0 from moving 0
(0.06) (2) DogBed_14b313d5 is blocking Agent 0 from moving 0
(0.06) (2) Chair_a080fe35 is blocking Agent 0 from moving 0
(0.06) (2) Sofa_4d86a6f3 is blocking Agent 0 from moving 0
(0.06) (2) DiningTable_38a37b8e is blocking Agent 0 from moving 0
(0.06) (2) DiningTable_9a2975d0 is blocking Agent 0 from moving 0
(0.03) (1) Chair_6d0d1366 is blocking Agent 0 from moving 0
(0.03) (1) Newspaper_cc1374a9 is blocking Agent 0 from moving 0
(0.03) (1) Book_3d15d052 is blocking Agent 0 from moving 0
(0.03) (1) Box_be6c27b5 is blocking Agent 0 from moving 0
(0.03) (1) DiningTable_64a5b5f3 is blocking Agent 0 from moving 0
(0.03) (1) ArmChair_705f4f83 is blocking Agent 0 from moving 0
(0.03) (1) Stool_3d5272a9 is blocking Agent 0 from moving 0
(0.03) (1) ArmChair_df5387ea is blocking Agent 0 from moving 0
%%%%%%%%%%

[MohitShridhar]

Ah yes, this navigation failure happens often. The A* navigator use a primitive grid-map representation to plan, so it's not very robust to different sized collision boxes of objects in the Unity scene.

[jzhanson]

I checked out this branch and tried building + running the docker, but there's what looks like a torchvision compatibility issue (the docker still installs Python 3.5.2, torch 1.5.1, and torchvision 0.6.1). I can't pip3 install --upgrade torch or torchvision because of the python version being < 3.6 (I'm still figuring out how to get the dockerfile or scripts/install_deps.sh to install a newer Python version, so any help on that front would be appreciated as well). Error text is below

Traceback (most recent call last):
  File "models/eval/eval_seq2seq.py", line 9, in <module>
    from eval_task import EvalTask
  File "/home/jzhanson/alfred/models/eval/eval_task.py", line 6, in <module>
    from eval import Eval
  File "/home/jzhanson/alfred/models/eval/eval.py", line 7, in <module>
    from models.nn.resnet import Resnet
  File "/home/jzhanson/alfred/models/nn/resnet.py", line 3, in <module>
    from torchvision import models, transforms
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torchvision/__init__.py", line 3, in <module>
    from torchvision import models
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torchvision/models/__init__.py", line 12, in <module>
    from . import detection
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torchvision/models/detection/faster_rcnn.py", line 14, in <module>
    from .roi_heads import RoIHeads
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torchvision/models/detection/roi_heads.py", line 210, in <module>
    @torch.jit.script
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/__init__.py", line 1290, in script
    fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/_recursive.py", line 568, in try_compile_fn
    return torch.jit.script(fn, _rcb=rcb)
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/__init__.py", line 1290, in script
    fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/__init__.py", line 2030, in _get_overloads
    compiled_fns.append(_compile_function_with_overload(overload_fn, qual_name, obj))
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/__init__.py", line 2010, in _compile_function_with_overload
    overload_signature = torch.jit.annotations.get_signature(overload_fn, None, None, inspect.ismethod(overload_fn))
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/annotations.py", line 79, in get_signature
    signature = parse_type_line(type_line, rcb, loc)
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/annotations.py", line 165, in parse_type_line
    arg_types = [ann_to_type(ann, loc) for ann in arg_ann]
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/annotations.py", line 165, in <listcomp>
    arg_types = [ann_to_type(ann, loc) for ann in arg_ann]
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/annotations.py", line 303, in ann_to_type
    the_type = try_ann_to_type(ann, loc)
  File "/home/jzhanson/alfred_env/lib/python3.5/site-packages/torch/jit/annotations.py", line 296, in try_ann_to_type
    the_type = torch._C._resolve_type_from_object(ann, loc, fake_rcb)
TypeError: _resolve_type_from_object(): incompatible function arguments. The following argument types are supported:
    1. (arg0: object, arg1: torch._C._jit_tree_views.SourceRange, arg2: Callable[[str], function]) -> torch._C.Type

I also have not started any x servers in side the docker (i.e. haven't run python3 startx.py &).

[MohitShridhar]

I'll keep this PR open to anyone interested in using 2.4.0 (or higher) for generating new trajectories. Thanks @jiasenlu for looking into this! Feel free to push your latest updates.

Unfortunately, we can't merge this into master because the language annotations were collected with 2.1.0. Minor changes to the scene layouts between 2.1.0 and 2.4.0 could invalidate some referring expressions and instructions in the dataset.

[oykuselin]

While following the directions properly, I couldn't run the code properly and got this error while running:

action is a string variable so in order to reach what is inside it the notation should be something like this action[0]/action[1] however inside code it is tried to reach this way: action['action'] which causes error:

Am I missing something or maybe code should be revised?