Zoom option

[thomassajot]

It is sometime hard to label small features. An option too zoom in would be useful.

If this feature is of interest, I made a partially functioning mock-up. It is missing the croping and scaling of the image, and how to handle non-square images. Optionally the full image can also be displayed in the smaller canvas on the right.

On the left if the image to label, on the right are the option to zoom in and out, and a display of where the zoomed image is with regards to the full scale image.

Example code

from ipycanvas import MultiCanvas
from ipywidgets import Button, VBox, HBox, Layout, Text
import numpy as np

class RectangleFlow(Canvas):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.prev_x = None
        self.prev_y = None
        self.fill_style = 'black'
        self.on_mouse_move(self.move_rect)
        self.rect_size = 10
        self.draw_contour()
    
    def clear_current_rect(self):
        if self.prev_x is not None:
            self.clear_rect(self.prev_x - (self.rect_size + 1), 
                            self.prev_y - (self.rect_size + 1), 
                            2 * (self.rect_size + 1),
                            2 * (self.rect_size + 1))
        self.draw_contour()
    def draw_contour(self):
        self.stroke_rect(0,0, self.width, self.height)
        
    def set_rect_size(self, value):
        self.clear_current_rect()
        self.rect_size = int(value)
        if self.prev_x is not None:
            self.move_rect(self.prev_x, self.prev_y)
            
    def move_rect(self, x, y):
        self.clear_current_rect()
        x = max(self.rect_size, min(self.width - self.rect_size, x))
        y = max(self.rect_size, min(self.height - self.rect_size, y))
        self.prev_x = x
        self.prev_y = y
        self.stroke_rect(self.prev_x - self.rect_size, 
                         self.prev_y - self.rect_size,
                         2 * self.rect_size, 
                         2 * self.rect_size)

class ZoomableCanvas(HBox):
    def __init__(self, image: np.ndarray):
        self.image = image
        
        self.canvas = Canvas(width=image.shape[0], height=image.shape[1])
        self.canvas.put_image_data(self.image, 0, 0)
        
        self.zoom_scale = 1
        self.zoom_info = Text(f'{self.zoom_scale * 100}%', layout=Layout(width='90px', height='30px'))
        self.zoom_btn = Button(description='+', layout=Layout(width='30px', height='30px'))
        self.unzoom_btn = Button(description='-', layout=Layout(width='30px', height='30px'))
        self.zoom_btn.on_click(self.zoom)
        self.unzoom_btn.on_click(self.unzoom)
        self.zoom_canvas = RectangleFlow(width=200, height=200)
        self.zoom_canvas.set_rect_size(self.zoom_scale * (self.zoom_canvas.width // 2))
        
        self.controls = HBox([self.zoom_btn, self.unzoom_btn, self.zoom_info])
        super().__init__()
        self.children = [self.canvas, VBox([self.controls, self.zoom_canvas])]
        self.draw_contour()
        
    def draw_contour(self):
        self.canvas.stroke_rect(0,0, self.canvas.width, self.canvas.height)
        
    def zoom(self, *args):
        self.zoom_scale -= 0.1
        self._update_zoom()
        
    def unzoom(self, *args):
        self.zoom_scale += 0.1
        self._update_zoom()
    
    def _update_zoom(self):
        self.zoom_scale = round(self.zoom_scale, 2)
        self.zoom_info.value = f'{self.zoom_scale * 100}%'
        self.zoom_canvas.set_rect_size(self.zoom_scale * (self.zoom_canvas.width // 2))
        zoomed_height, zoomed_width = self.zoom_scale * np.array(self.image.shape[:2])
        self.canvas.clear()
        self.canvas.put_image_data(self.image[0:int(zoomed_height), 0:int(zoomed_width)])
        self.draw_contour()
       

x = np.linspace(-1, 1, 300)
y = np.linspace(-1, 1, 300)

x_grid, y_grid = np.meshgrid(x, y)

blue_channel = np.array(np.sin(x_grid**2 + y_grid**2) * 255, dtype=np.int32)
red_channel = np.zeros_like(blue_channel) + 200
green_channel = np.zeros_like(blue_channel) + 50

image_data = np.stack((red_channel, blue_channel, green_channel), axis=2)
ZoomableCanvas(image_data)

[janfreyberg]

This is a really great idea and I had thought this would be nice-to-have but wasn't sure how it would be implemented. This is great and I like the user interface with the zoom buttons.

However, at the moment It doesn't seem to actually zoom - it just crops?

Also, when I put the mouse inside the zoom canvas, I get a lot of jittering - I wonder if this is because the program makes a callback every time the mouse moves. Maybe clicking and dragging would be a bit better on the zoom_canvas?

[thomassajot]

This was a quick and dirty attempt.
I need to add the scaling option (using open cv2 or skimage, or something else if there are any suggestions).
The mouse is the hardest one. Ipycanvas does not get mouse clicks (or scroll), so it is not possible to drag and drop the window (or zoom in/out). I was thinking of binding a key to enable or disable the mouse tracking within the canvas. However it might not be smooth enough. In addition, it is possible to add 4 buttons (up down left and right) to manually move the window by a set amount of pixel.

[thomassajot]

Code Update, the functionality is there but it needs improvements.

from ipycanvas import Canvas
from ipywidgets import Button, VBox, HBox, Layout, Text
import numpy as np
from skimage.transform import resize, rescale


class RectangleFlow(Canvas):
    """Canvas which contains a square which follows the mouse cursor"""
    def __init__(self, image, rect_size=10, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if image is not None:
            self.bg_img = resize(image, output_shape=[self.height, self.width], preserve_range=True)
        else:
            self.bg_img = None

        self.rect_size = rect_size
        # position rectangle in the center of the canvas
        self.x = (self.width // 2) - rect_size
        self.y = (self.height // 2) - rect_size
        
        self.fill_style = 'black'
        self.on_mouse_move(self.move_rect)
        self.draw_image()
        self.draw_contour()
        self.draw_current_rect()
        self.clear_current_rect()
    
    def clear_current_rect(self):
        margin = 2
        self.clear_rect(self.x - margin, 
                        self.y - margin, 
                        self.rect_size + 2 * margin, 
                        self.rect_size + 2 * margin)
        self.draw_image(self.x - margin, 
                        self.y - margin, 
                        self.rect_size + 2 * margin, 
                        self.rect_size + 2 * margin)
        self.draw_contour()
        
    def draw_current_rect(self):
        self.stroke_rect(self.x, self.y, self.rect_size, self.rect_size)
        
    def draw_contour(self):
        self.stroke_rect(0, 0, self.width, self.height)
        
    def draw_image(self, x=None, y=None, width=None, height=None):
        if self.bg_img is None:
            return 
        
        if x is None:
            x = 0
            
        if y is None:
            y = 0
            
        if width is None:
            width = self.width
            
        if height is None:
            height = self.height
        y = max(0, min(self.bg_img.shape[0], y))
        x = max(0, min(self.bg_img.shape[1], x))
        self.put_image_data(self.bg_img[y : y + height, x : x + width], x, y)
        
    def set_rect_size(self, value):
        self.clear_current_rect()
        self.rect_size = int(value)
        self.move_rect(self.x, self.y)
            
    def move_rect(self, x, y):
        # the cursor (x, y) is at the center of the rectangle
        self.clear_current_rect()
        x = x - self.rect_size // 2
        y = y - self.rect_size // 2
        self.x = int(max(0, min(self.width - self.rect_size, x)))
        self.y = int(max(0, min(self.height - self.rect_size, y)))
        self.draw_current_rect()

class ZoomableCanvas(HBox):
    def __init__(self, image: np.ndarray):
        self.image = image
        self.zoomed_image = image
        # canvas displaying the zoomed image
        self.canvas = Canvas(width=image.shape[0] + 2, height=image.shape[1] + 2)
        self.canvas.stroke_rect(0,0, self.canvas.width, self.canvas.height)
        
        # zoom config
        self.zoom_scale = 1
        self.zoom_info = Text(f'{self.zoom_scale * 100}%', layout=Layout(width='90px', height='30px'))
        self.zoom_btn = Button(description='+', layout=Layout(width='30px', height='30px'))
        self.unzoom_btn = Button(description='-', layout=Layout(width='30px', height='30px'))
        self.zoom_btn.on_click(self.zoom)
        self.unzoom_btn.on_click(self.unzoom)
        self.zoom_canvas = RectangleFlow(image, rect_size=200, width=200, height=200)
        self.zoom_canvas.on_mouse_move(self._update_zoom_position)
        
        self.zoom_controls = HBox([self.zoom_btn, self.unzoom_btn, self.zoom_info])
        
        super().__init__()
        self.children = [self.canvas, VBox([self.zoom_controls, self.zoom_canvas])]
        self._update_canvas()
        
    def zoom(self, *args):
        self.zoom_scale = max(0.1, self.zoom_scale - 0.1)
        self._update_zoom()
        
    def unzoom(self, *args):
        self.zoom_scale = min(1., self.zoom_scale + 0.1)
        self._update_zoom()
    
    def _update_zoom(self):
        self.zoom_scale = round(self.zoom_scale, 2)
        self.zoom_info.value = f'{self.zoom_scale * 100:.1f}%'
        self.zoom_canvas.set_rect_size(self.zoom_scale * self.zoom_canvas.width)
        self._update_zoom_position()
        
    def _update_canvas(self):
        self.canvas.put_image_data(self.zoomed_image, 1, 1)
        
    def _update_zoom_position(self, *args):
        # this is the x/y position of the cursor
        # we want the x/y position of the zoom_canvas inner rectangle
        height = int(self.image.shape[0] * self.zoom_scale)
        width = int(self.image.shape[1] * self.zoom_scale)
        y = int(self.image.shape[0] * (self.zoom_canvas.y / self.zoom_canvas.height))
        x = int(self.image.shape[1] * (self.zoom_canvas.x / self.zoom_canvas.width))
        
        self.zoomed_image = resize(self.image[y : y + height, x : x + width], 
                                   output_shape=[self.canvas.height - 2, self.canvas.width - 2], 
                                   preserve_range=True)
        self._update_canvas()
        
       

x = np.linspace(-1, 1, 500)
y = np.linspace(-1, 1, 500)

x_grid, y_grid = np.meshgrid(x, y)

blue_channel = np.array(np.sin(x_grid**2 + y_grid**2) * 255, dtype=np.int32)
red_channel = np.zeros_like(blue_channel) + 200
green_channel = np.zeros_like(blue_channel) + 50

image_data = np.stack((red_channel, blue_channel, green_channel), axis=2)
z = ZoomableCanvas(image_data)
z

I got some issues with the refresh rate. The functionality is there but the update is way too slow.

Any idea why ?

[thomassajot]

New version working much better but there is still some lag due to the canvas.put_image_data function (I believe).
The new version supports rectangle images. The zoom box supports mouse drag.

[janfreyberg]

Thanks! That looks really nice. Would you be able to open a pull request so I can try it out?

It's worth thinking about how the data will get translated between the zoomed in canvas - the pixel locations of polygon points still need to be relative to the overall image I think...

[thomassajot]

I was thinking about it.
One way to do it is to modify the AbstractAnnotationCanvas and wrap the methods on_click, on_drag and on_release to modify the (x, y) coordinates of the mouse from the canvas space to the transformed space.
There would be no changes to the Child of this class which is nice.

And then the rest would be done in the Annotator class.

[janfreyberg]

Yes, that sounds like a good approach.

Having thought about it in this context I realise now that I think there's a bug in the library right now - I think at the moment, if the image is bigger than the canvas, I shrink the image, but the x/y coordinates of the annotations are recorded relative to the canvas only...

This can probably be fixed with a solution that also works for zoomed-in canvases; maybe by implementing a function like correct_coordinates that works in both cases.

Thanks for thinking about this. Do you want to have a go at making that conversion function? Otherwise I can have a think about it over the weekend.

[thomassajot]

Good catch.
I am not so happy about the approach yet.
Modifying the Abstract class would make it have more functionality that only gets used in the zoomed setting. But I did not find a better solution so far. I might still implement it today to see how bad it is.

But ! With your big to solve; maybe it’s worth the effort ? As in, the Abstract class would be given the x, y coordinates. As it holds the load_image, transformations can be applied there, and also on the x,y coordinates of the functions call backs.

[thomassajot]

A new Pull Request has been created.
There is a notebook with a working example for Poly, box and Point annotations.
The zoom is functional, but the translation is not implemented fully yet.
Feel free to point out any issues.

[thomassajot]

The shapes are custom classes. You may be interested by the Shapely library which handles geometries and a wide variety of functionality on top of it. A useful one is the serialisation of the objects.