Occlusion is not implemented for multimesh prims

[franz6ko]

Description

After several tests using the PoseWriter, I found that the visibility field (based on the occlutionRatio field from the bounding_box_3d annotator) is only working for very simple assets and not for multimesh assets. This is very critical because most of the most used assets (such as human actors) are multimesh assets.

The issue is very simple to reproduce opening an empty world and running the following code using the script editor

import os
from datetime import datetime

import omni.replicator.core as rep

# Build output path
base_output_dir = (
    "/home/ubuntu/ws/src/sandbox/_output"
)
datetime_folder = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
output_dir = os.path.join(base_output_dir, datetime_folder)

with rep.new_layer():
    # Create objects
    PERSON_SRC =  "https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1/Isaac/People/Characters/female_adult_police_03_new/female_adult_police_03_new.usd"
    person = rep.create.from_usd(PERSON_SRC, semantics=[('class', 'person')])

    # Randomize positions
    with rep.trigger.on_frame(max_execs=10):
        pass

    # Create camera and render product
    camera = rep.create.camera(position=(14, 14, 14), look_at=(0, 0, 0))
    render_product = rep.create.render_product(camera, (1280, 720))

    # Initialize and attach writer
    writer = rep.WriterRegistry.get("PoseWriter")
    writer.initialize(
        output_dir=output_dir + "/pose_writer",
        visibility_threshold=-1.0,
        skip_empty_frames=False,
        write_debug_images=True,
    )
    writer.attach([render_product])
    rep.orchestrator.preview()

    # Basic writer
    basic_writer = rep.WriterRegistry.get("BasicWriter")
    basic_writer.initialize(
        output_dir=output_dir + "/basic_writer",
        bounding_box_2d_tight=True,
    )
    basic_writer.attach([render_product])

    rep.orchestrator.preview()

This seems to be a known issue for more than a year now in this discussion

I am open to provide more details or help.

Isaac Sim version

5.1.0

Operating System (OS)

Ubuntu 24.04

GPU Name

RTX 6000 Ada

GPU Driver and CUDA versions

Driver 550.163.01, CUDA 12.4

Logs

No response

Additional information

No response

[RandomOakForest]

Thank you for posting this. This is a known limitation. You need a mechanism that aggregates occlusion (and thus, visibility) across all sub-meshes that make up your logical asset. NVIDIA Isaac Sim supports creating custom annotators. You can build an annotator based on bounding_box_3d that iterates over all child meshes of a given asset, calculates occlusion per mesh, and aggregates the results for the parent asset.

Example approach (conceptual, not full code):

# Pseudocode: Aggregates occlusion ratios for all meshes of each asset
occlusion_ratios = []
for mesh in asset_meshes:
    ratio = compute_occlusion(mesh)
    occlusion_ratios.append(ratio)
# Aggregate, for example:
visibility = max(occlusion_ratios)  # Or use mean, as fits your use case

Please submit a feature request if you have a specific use case we can pursue. Thank you.

[franz6ko]

Hi @RandomOakForest, thanks for your answer.

I think that more than specific use of case, this is something that must be limiting a lot of applications as many assets are multimesh (like human actors for example).

Is there any known issue or limitation about implementing the approach you explain ? I imagine that if this is quite trivial then it should be already done. So I imagine there are more limitations behind. If this is not the case, do you know if there is some documentation or example on how to write a custom annotator and make it work ?

Thanks in advance

[RandomOakForest]

Following up, see for example this doc on annotators. I'll move this post to our Discussions for follow up. Thank you for your interest in Isaac Sim.