How to Set Up Image Registration in Python

In digital image analysis or processing, image registration is the process of aligning or comparing two or more images of the same scene taken at different times or from different viewpoints, in order to transform the image data into a single coordinate system.

The aim of an image registration algorithm is to transform a moving image so that it is spatially or temporally aligned with a fixed target image. It’s often divided into two parts: the sort of modification that can be conducted on the moving picture (the transformation model) and a specification of alignment (the similarity cost function) between the images.

To date, several techniques have been developed to solve the problem of image registration, including traditional methods like spatial transformation and deep-learning based models such as Convolutional Neural Networks (CNNs).

In this article, we’ll dive into the details of image registration, how it works and in the end we’ll build an example implementation of image registration using the OpenCV library in Python.

In this article:

• Why Do We Need Image Registration?
• How Does Image Registration Work?
• Image Registration in The Real World
• Image Registration in Python Using OpenCV

Why Do We Need Image Registration?

Image registration is necessary when we have two images of the same scene or object, but they do not perfectly overlap. This often happens if something moves between the two photos or if the camera angle changes slightly.

For example, imagine taking two pictures of a person. If either the person or the camera shifts between the photos, the person might appear in different positions in each picture. Likewise, if you zoom in on the second photo, their face might take up more space in the frame, and the background might also look different.

The purpose of image registration is to align these two images so that the overlapping parts match up as closely as possible. This is useful in many areas, especially in medical imaging for diagnostic, therapeutic, and research purposes.

How Does Image Registration Work?

A typical image registration task works by utilizing the following key concepts:

• Feature detection: This is usually the first step in image registration. Here, the goal is to identify distinctive points or regions in the images that can be easily recognized and matched. Common types of features include:
  • Corners (points where two edges meet)
  • Edges (boundaries between different regions in the image)
  • Blobs (regions of interest that differ in properties like brightness or color compared to surrounding areas)
• Feature matching: Once features are detected, the next step is to find corresponding features across the images. This is typically done by comparing feature descriptors, which are numerical representations of the features and their surrounding areas. Example algorithms used for this step include ORB (Oriented FAST and Rotated BRIEF) and SIFT (Scale-Invariant Feature Transform).
• Transformation model estimation: After matching features, the next step is to determine the geometric transformation that will align one image with the other. The transformation may be rigid (translation, rotation, reflection), affine (translation, rotation, scaling, reflection, shearing), projective or nonlinear.
• Image transformation: Once transformation is done, it’s then applied to one image (usually the “sensed” image) to align it with the reference image. This process involves mapping each pixel from the original image to its new location in the transformed image.

Although there are several image registration algorithms that have been development, the most common algorithm involves taking a moving image and transforming it to be spatially or temporally aligned with a target fixed image.

Another algorithm is based on intensity-based registration. This method uses the pixel values (intensities) of the images to align them to find the transformation that maximizes the similarity between the pixel values of the source and the target images. This method usually works well when there are no distinctive features to track in the images.

Image Registration in The Real World

Computer vision

In computer vision, image registration is used in object tracking by aligning images or frames in videos to track moving objects. Similarly, in 3D reconstruction, multiple 2D images from different viewpoints can be registered to create 3D models of objects or scenes. Other applications of image registration in computer vision include motion detection and aligning virtual objects with real-world scenes in augmented reality applications.

Medical Imaging

In medical imaging, techniques such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) scans are used to visualize internal organs, tissues and anatomy of patients. For example, spatial normalization, a method of mage registration, is used in MRI scans of human brains to establish correspondence between brain scans so that a single structure in one subject’s brain scan can be compared to the same area in another subject’s scan.

Panoramic images

This is one area you may be familiar with where image registration is commonly used. In panoramic imaging, image registration is used to create wide-angle or 360-degree panoramic views from multiple images. On smartphones, it is used to stitch together different photos into a single landscape image and, in manufacturing; it enables an enhanced field of view for robotics and autonomous systems.

Military surveillance

Military surveillance applies image registration for automatic target recognition and tracking, detecting changes in satellite or aerial imagery, fusing data from multiple sensors to improve situational awareness, mapping and analyzing terrain.

Digital image analysis

Image analysis techniques, such as georeferencing and digital image correlation/tracking use image registration methods for analyzing images in different contexts. Georeferencing is a geographic form of image registration that assigns real-world geographic coordinates to an image, such as aerial or satellite images, to Geographic Information Systems (GIS).

Image Registration in Python Using OpenCV

In this section, we’ll walk you through an example of image registration in Python using OpenCV (for image processing), Numpy (for numerical operations), and Matplotlib (for plotting and visualizing our images).

Let’s take the following images, for example.

You can download the images below:

• Original image
• Rotated image

Above, we have two images of the same person, but the one on the right is rotated and misaligned with the one on the left. Our goal is to align/register the two images into a single image. Here’s how to do that:

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Load the original image
img_original = cv2.imread('headshot-one.jpg', cv2.IMREAD_GRAYSCALE)

# Load the shifted/rotated image
img_shifted = cv2.imread('headshot-two.png', cv2.IMREAD_GRAYSCALE)

# Ensure both images are of the same size
img_original = cv2.resize(img_original, (img_shifted.shape[1], img_shifted.shape[0]))

# Display the original and shifted images for reference
plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.imshow(img_original, cmap='gray')
plt.title('Original Image')

plt.subplot(1, 2, 2)
plt.imshow(img_shifted, cmap='gray')
plt.title('Shifted/Rotated Image')
plt.savefig('output_image.png')

# Define the motion model: We'll use an affine transformation
warp_mode = cv2.MOTION_AFFINE

# Set up the transformation matrix, initially set to identity
if warp_mode == cv2.MOTION_HOMOGRAPHY:
    warp_matrix = np.eye(3, 3, dtype=np.float32)
else:
    warp_matrix = np.eye(2, 3, dtype=np.float32)

# Set the termination criteria: max iterations or convergence epsilon
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 5000, 1e-10)

# Apply the ECC algorithm to find the warp matrix (alignment transformation)
(cc, warp_matrix) = cv2.findTransformECC(img_original, img_shifted, warp_matrix, warp_mode, criteria)

# Warp the shifted image back to align with the original image
if warp_mode == cv2.MOTION_HOMOGRAPHY:
    # Use warpPerspective for Homography
    img_aligned = cv2.warpPerspective(img_shifted, warp_matrix, (img_original.shape[1], img_original.shape[0]),
                                      flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP)
else:
    # Use warpAffine for other transformation models
    img_aligned = cv2.warpAffine(img_shifted, warp_matrix, (img_original.shape[1], img_original.shape[0]),
                                 flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP)

# Display the aligned image alongside the original
plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.imshow(img_original, cmap='gray')
plt.title('Original Image')

plt.subplot(1, 2, 2)
plt.imshow(img_aligned, cmap='gray')
plt.title('Aligned Image')
plt.savefig("registered_image.png")

# Calculate the Mean Squared Error (MSE) between the original and aligned images
def mse(image1, image2):
    """Compute the mean squared error between two images"""
    err = np.sum((image1.astype("float") - image2.astype("float")) ** 2)
    err /= float(image1.shape[0] * image1.shape[1])
    return err

error = mse(img_original, img_aligned)
print(f"Mean Squared Error after alignment: {error}")

Running the code gives the following output:

The alignment also returned an MSE value of 13.68691224489796, meaning that the pixel values of the two images are quite close to each other and the registration (alignment) process worked well, as the aligned image is very similar to the original.

Let’s take a look at another example where the two are very different:

Here are the sources for the images:

• Original image
• Rotated image

Running the code using the images above return the following image output:

The alignment returned a MSE value of 2078.064832022596, which is quite high, meaning there’s a large difference between the two images and the registration process was not effective, or the images might not be properly aligned.

Conclusion

In conclusion, image registration is a powerful technique in fields that rely on precise image analysis, allowing processing of different types of visual data efficiently. In this article, we’ve explored how to implement image registration using Python, specifically with the OpenCV library.However, other libraries like scikit-image or MATLAB also provide functionalities for image registration through feature-based or intensity-based methods (e.g., using phase cross-correlation).