Add CorrespondenceCheckerBasedOnRotation for orientation-constrained RANSAC

[jGiltinan]

Type

• Bug fix (non-breaking change which fixes an issue): Fixes #
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

In structured environments, such as an overhead laser profiler mounted on an X-Y gantry or fixed above a conveyor belt, objects rest on an approximately flat surface. Their rotation about the Z-axis is unknown, but their orientation about the X and Y axes is physically constrained to within a small tolerance by gravity and the surface they rest on.

Existing Open3D correspondence checkers operate purely on local point geometry (edge lengths, point-to-point distance, surface normals). They have no mechanism to encode this kind of global orientation prior. As a result, RANSAC wastes iterations validating physically impossible, tilted or inverted orientations, and can become trapped in catastrophically wrong local minima, producing correct-looking local geometry fits that are globally flipped or upside-down which causes ICP to diverge.

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

This PR introduces CorrespondenceCheckerBasedOnRotation, a new correspondence checker that allows developers to independently constrain the allowed rotation of a RANSAC hypothesis about any of the three Cartesian axes (X, Y, Z). Each axis is configured with a tolerance in radians. A value of -1 (the default) disables the check for that axis, making the checker flexible for a wide variety of structured scanning applications.
Example use:

checker = o3d.pipelines.registration.CorrespondenceCheckerBasedOnRotation(
   [np.pi * 10 / 180, # x-axis constrained to +/- 10 degrees
    np.pi * 10 / 180, # x-axis constrained to +/- 10 degrees
    1  # z-axis is unconstrained
    ]
)

The motivating case, along with sample results is summarized in a video: https://youtu.be/VuRbrNXATr8

For the benchmark results, the checker was tested on 3DBenchy registered against a synthetic laser scan, over 5 deterministic random seeds, single-threaded for determinism.

Configuration	Mean Total Time	Std Dev	Registration Failures
Current Open3D (EdgeLength + Distance + Normal)	139s	±157s	1 in 5
CorrespondenceCheckerBasedOnRotation only	603s	±55s	0 in 5
Combined (Rotation first, then EdgeLength + Distance + Normal)	90s	±19s	0 in 5

Key observations:

• Current checkers alone are fast per iteration but fail on symmetric/ambiguous geometry by converging to a physically impossible (inverted) alignment (Seed 43).
• The rotation checker alone is perfectly robust but slow, as it does not constrain translation, allowing RANSAC to waste expensive validation steps on bad-translation hypotheses.
• Combined, the rotation checker cheaply eliminates impossible global orientations early (fail-fast), and the vanilla checkers constrain local geometry. The result is both faster and perfectly robust across all seeds.

The code for the benchmark results is located at: https://github.com/jGiltinan/Open3D-Demonstrations

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[Copilot]

Pull request overview

This PR adds a new RANSAC correspondence checker that rejects hypotheses whose estimated rotation exceeds per-axis tolerances, enabling orientation priors for structured scanning/registration scenarios.

Changes:

• Introduces CorrespondenceCheckerBasedOnSourceRotation in C++ and exposes it via pybind.
• Adds Python-facing API/docs and __repr__ for configuring per-axis rotation thresholds.
• Updates RANSAC documentation to list the new checker and adds a changelog entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
cpp/pybind/pipelines/registration/registration.cpp	Binds the new correspondence checker into the Python registration module and updates parameter docs.
cpp/open3d/pipelines/registration/CorrespondenceChecker.h	Declares the new checker class and its public API/docs.
cpp/open3d/pipelines/registration/CorrespondenceChecker.cpp	Implements the rotation-threshold check against the estimated transform.
CHANGELOG.md	Notes the addition of the new rotation-based checker.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jGiltinan]

I have resolved the comments on documentation, thank you for running those checks. As those were straightforward, I marked them as resolved.

I did mark the comment on the euler angle vs angle-axis rotation as resolved, as it was resolved to my idea of the checker. However, it is important for the reviewer to review that section to make sure the intended functionality is clear and is appropriate for merging into the library.