Update PointCloud type to support client-side rendering

[DTCurrie]

This change introduces a PointCloudHeader to the PointCloud message, making the existing bytes payload self-describing. This change is additive, non-breaking, and significantly improves our ability to describe and render point clouds.

Reasoning

The current PointCloud message efficiently transports data as a binary blob (bytes point_cloud = 1;). However, it lacks any descriptive metadata, forcing an implicit, out-of-band contract between the client and server to interpret the byte stream's structure (e.g., field order, data types, presence of color or intensity).

This approach has three main drawbacks:

• Brittleness: Any change to the data layout on the server requires all clients to be updated in lockstep to avoid parsing errors.
• Lack of Extensibility: It is difficult to add new per-point attributes (like normals or classification labels) in a way that is discoverable and manageable for API consumers.
• Ambiguity There is no way to describe if a point cloud includes additional data like colors without relying on values from other messages like Transform.metadata.

Proposed Changes

To address this, I introduced the PointCloudHeader message. This header adheres to industry standards to explicitly define the structure of the data contained within the point_cloud byte array.

I updated the PointCloud message with a new field, PointCloudHeader header = 2;, to contain the metadata.

This change is non-breaking. Existing clients will continue to parse the bytes at field 1 as they always have and will ignore the new header field at tag 2. New clients can check for the presence of the header to adopt a more robust parsing logic, enabling them to handle varied and evolving point cloud structures.

Rendering Benefits

This enhancement directly benefits rendering performance by allowing the server to provide a GPU-optimized, interleaved memory layout. In this Array-of-Structures format, all attributes for a single point (e.g., position, color, intensity) are packed contiguously in the binary blob. A rendering client using a library like Three.js can leverage this for maximum efficiency. The entire bytes payload from the message is loaded into a single TypedArray and used to create a THREE.InterleavedBuffer.

The new PointCloudHeader provides the crucial stride value—the total number of elements for one point—needed for this step. From this single buffer, visualizers can create multiple THREE.InterleavedBufferAttribute's to define 'position', 'color', and other properties by specifying their individual itemSizeandoffset` within the stride, all of which are now explicitly defined in the header. This "zero-copy" process is speedy because it avoids any client-side iteration or data restructuring. The data flows directly from the network into a memory layout that is highly cache-friendly for the GPU, ensuring that when a vertex is processed, all of its associated attributes are fetched in a single memory read, maximizing rendering throughput for large-scale point clouds.

Beyond the initial transfer of a complete point cloud, this self-describing format is exceptionally well-suited for streaming targeted updates to a client. For instance, if only a small region of the point cloud changes (e.g., an object moves or a sensor updates a specific area), the server can send a new PointCloud message containing only the data for the modified points by including the start field in the header. Upon receiving this partial update, the client can perform a highly efficient, targeted modification of the data on the GPU.

Instead of rebuilding the entire geometry, the client can use the new binary data to overwrite a specific portion of its existing THREE.BufferAttribute or THREE.InterleavedBuffer. The client can inform the renderer to only re-upload the small, changed segment of the buffer to the GPU, and avoid the significant performance cost of transferring and processing the entire multi-million point dataset for a minor change.

Quick Example

I have already put together a working POC using the world state store service fake as an example. You can see the code here:

• RDK: Add PCD to WorldStateStore service fake with color visualization rdk#5311
• motion-tools: Updated point cloud rendering based on proposed API viam-labs/motion-tools#138

Screen.Recording.2025-10-06.at.9.28.46.AM.mov

[micheal-parks]

Why not a frame of the sensor that captured it? that's what we currently do with pointclouds in the visualizer. pointclouds are returned from camera methods in local space and we child them to the camera frame. it allows us to show both local and world pose information

[micheal-parks]

I guess though if this is just modifying the proto message then we can still determine the frame using the above strategy. would viewpoint here potentially mean some other kind of offset with respect to the sensor frame?

[DTCurrie]

We can do either, but I am thinking more from an ergonomics standpoint. I imagine it would be easier to work with just the pose than the entire frame object when using this API. I also feel like the Pose is less likely to change (at least in the near future) as opposed to the Frame.