Computer vision | Alan Hung

Investigation of Algorithms for Retrieving Similar Images

Tue, 10 Jul 2018 00:00:00 +0000

Table of Contents

Abstract

Automating the task of searching for similar images of fabric is crucial to improve efficiency in the fabric-related industry. Three algorithms were investigated in this project: per-pixel measures ($L_2$), K-Nearest Neighbors (KNN), and perceptual similarity (LPIPS). While LPIPS provides a better way to retrieve similar fabric as perceived by humans, its computational expense becomes a main obstacle when the number of images increases.

1. Introduction

Searching for similar images of fabric is a common task in the fabric-related industry. For instance, buyers may want to purchase fabrics similar to previous best-selling ones, while designers may want to experiment with different yet similar fabrics during the product design process.

Although humans can recognize similar images effortlessly, traditionally, this task has relied on human search through all available fabrics to find a similar one, which is time-consuming and exhausting. Automating this procedure is crucial to improve efficiency.

2. Search algorithm

Since we are searching through all images, we will use the notation $N$ to represent the total number of images in the following discussion.

2.1. Per-pixel measures ($L_2$ distance)

One of the classic methods for comparing images is to calculate the difference of every pixel using the $L_2$ distance. The time complexity is $O(N^2 * C * W * H)$, where $C$ represents the number of channels in the image, and $W$ and $H$ represent the width and height of the image, respectively. The space complexity is $O(N^2)$ because we need to store all the distances between the images.

2.2. K-Nearest Neighbors (KNN)

Rather than comparing the images directly, we can extract features from the image, such as the main color, color schemes, pattern size, and so on. Similarity can be defined as the $L_2$ distance between features. However, as the number of features increases, all samples may appear dissimilar, which is often referred to as the “curse of dimensionality”. To overcome this problem, we can reduce the dimensionality using PCA.

The time complexity is $O(N^2 * K)$, where $K$ represents the number of principal components from PCA. The space complexity remains $O(N^2)$ since we store all distances.

2.3. Perceptual similarity (LPIPS)

This method, which was published by Zhang ( Citation: Zhang, Isola & al., 2018 Zhang, R., Isola, P., Efros, A., Shechtman, E. & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595. ) as shown in the Fig. 1, extracts feature maps from activation layers and calculates the weighted distance $d_0$ between those feature maps. To apply this method, we can push images $x$ and $x_0$ through a pre-trained network (such as AlexNet ( Citation: Krizhevsky, 2014 Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997. ) ), denoted as $F$.

Fig. 2 provides an example with AlexNet. When the input images exhibit visual differences, as in Fig. 2 (a), the spatial average result differs significantly from that of the similar input images in Figure Fig. 2 (b).

While this method is proven to be effective in finding similar images, it is computationally expensive. Depending on the architecture of the pre-trained network, which may involve multiple convolutional and pooling layers, the runtime may be significantly increased. In addition, we will need additional memory to store all feature maps from activation layers.

3. Experiments

3.1. Dataset

We collected 113 digital images of fabric with dimensions of 30 cm x 42 cm (3500 x 4200 pixels), as shown in Fig. 3. To test the algorithm’s ability to retrieve similar images, we randomly select images from the dataset, add spatial pixel shifts, and then put them back into the dataset. We then test whether the algorithm can retrieve them.

3.2. Result

As shown in the Fig. 4, the use of per-pixel measures ($L_2$) may fail to retrieve images with spatial pixel shift. This is because even a small shift can result in a large difference. In contrast, KNN and LPIPS rely solely on extracted features and can therefore successfully find those images.

However, due to the lack of spatial information in features, KNN might still fail to find images that are similar from a human perspective. As shown in the Fig. 5, the KNN algorithm retrieves images that are similar to the input image only in terms of colors. However, the context and structure of these images are entirely different.

As previously discussed, in terms of runtime (as shown in the Fig. 6), LPIPS is the most computationally expensive of the three algorithms, taking up to 8.3 seconds per image on our dataset.

4. Application

To improve the efficiency of fabric searching, LPIPS with AlexNet has been implemented into the in-house library, as shown in the Fig. 7. When a new image is imported, the algorithm retrieves similar fabrics and stores them in a database.

5. Conclusion

Three algorithms were investigated in this project. The per-pixel measure ($L_2$) did not meet our requirements due to its sensitivity to pixel shift. Similarly, KNN is not applicable since the features lack spatial information. LPIPS provides a better way to retrieve similar fabric as perceived by humans, but its computational expense becomes a main obstacle when the number of images increases.

Reference

Zhang, Isola, Efros, Shechtman & Wang (2018): Zhang, R., Isola, P., Efros, A., Shechtman, E. & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595.
Krizhevsky (2014): Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997.

Repeated Pattern Detection of Printed Fabric

Thu, 01 Feb 2018 00:00:00 +0000

Table of Contents

Abstract

We study the problem of detecting repeated patterns. Our goal is to extract a single unit of repeated patterns from an input image. This problem can be simplified for printed fabric because of its strictly regular and fronto-parallel nature. Under these properties, we investigate two algorithms. The first one uses Fourier analysis, but it is not suitable for our purpose because it requires additional assumptions about the period of the image. The second algorithm involves shifting the image and calculating the distance using the $L_2$ loss function. This method can efficiently solve the problem compared to human processing on our dataset.

1. Introduction

Repeated patterns are prevalent, particularly in human-made objects, as illustrated in the Fig. 1 (a). They offer insights into the object, such as allowing us to estimate the number of floors and the structure of a building ( Citation: Lettry, Perdoch & al., 2017 Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55. ) or count the number of objects in an image ( Citation: Park, Brocklehurst & al., 2009 Park, M., Brocklehurst, K., Collins, R. & Liu, Y. (2009). Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10). 1804–1816. ) .

This is especially important for printed fabric in the textile industry, as it consists of repeated patterns in both the vertical and horizontal directions. Extraction a single pattern is useful in two ways. First, It provides buyers with the minimum quantity necessary to purchase without having an incomplete pattern. For example, one could consider purchasing two units in the vertical direction to have two birds in opposite direction in a single pattern, as shown in the Fig. 1 (b). Second, organizing a library by patterns can help designers find patterns more efficiently, ultimately speeding up the design process.

2. Main challenge

Automating repetition detection remains a challenging task since the definition is unclear. For example, a checkerboard consisting of black and white has perfect repetition. However, it is often accompanied by irregularities, as shown in the Fig. 1 (a), the windows of the building are similar to each other, rather than being exactly the same in each pattern.

Several algorithms have been developed to address this deviation ( Citation: Liu, Collins & al., 2004 Liu, Y., Collins, R. & Tsin, Y. (2004). A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE transactions on pattern analysis and machine intelligence, 26(3). 354–371. ; Citation: Liu & Liu, 2013 Liu, J. & Liu, Y. (2013). Grasp recurring patterns from a single view. Proceedings of the IEEE conference on computer vision and pattern recognition. 2003–2010. ) , while others make assumptions with a nearly regular pattern ( Citation: Lettry, Perdoch & al., 2017 Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55. ) . In other words, assumptions are crucial for implementing an algorithm.

3. Program

Although it is a complex problem, fortunately, for our purposes, it is limited to printed fabric, which is both strictly regular and fronto-parallel due to its manufacturing process. As a result, our problem is as simple as detecting the perfect repetition of a checkerboard. The problem’s constraint depends on the resolution of the image, which in our case is 3500 x 4200 pixels.

Constraints:

len(pixels) = 3500 x 4200

In the following analysis, we will use the notation $N$ to represent the total number of pixels in an image.

3.1. Approach with Fourier analysis

The intuitive method for identifying the period would be to use Fourier analysis, which is made efficient by using FFT. This results in a time complexity of $O(NlogN)$. However, since the basic idea behind Fourier analysis is an approximation by sums of simpler trigonometric functions, identifying the peak in spectrum can be challenging.

For instance, let’s say we perform Fourier analysis in the horizontal direction of a checkerboard. As shown in the Fig. 2 (a), the frequency of the maximum power corresponds to the pattern consisting of a pair of black and white squares. If we reduce the contrast, as also depicted in the Fig. 2 (b), humans can still effortlessly identify the pattern. However, the position of the maximum power in the spectrum shifts to a lower frequency. In other words, to find the same pattern as that perceived by humans, we need to specify the search range in the spectrum. Nevertheless, since there is no certain range of periodicity for images, this approach is not practical.

3.2. Approach with $L_2$ loss

This approach mimics human processing by sliding images to find their periodicity. In other words, it compares the pixels of images one by one, and uses the $L_2$ loss function as a metric for the distance between images. Here is how the algorithm works, as shown in the Fig. 3:

Create a copy of the original image.
Fixed one image as a reference and slide another.
- Calculate the loss in each RGB channel.
- Calculate the average loss for overlapped regions.
Calculate the midpoint of the average loss function.
Measure the distance between two midpoints while ascending or descending.

The size of the pattern can be extracted by performing the algorithm in both horizontal and vertical directions. The algorithm’s time complexity is $O(N^2)$ since we need to iterate through all pixels in each iteration. The space complexity is $O(N)$ because we create a copy of the original image.

Although the algorithm is $N^2$, all elements share the same computation at every step. Therefore, we can take advantage of vectorization by using Numpy. Additionally, it is necessary to resize the image for the algorithm, which involves a trade-off between runtime and accuracy.

4. Experiment

4.1. Dataset

To test the speed and accuracy of our algorithm, we collected 113 digital images of fabric with dimensions of 30 cm x 42 cm (3500 x 4200 pixels), as shown in Fig. 4 (a). The pattern size was extracted manually using the same method as the algorithm, which involved shifting and comparing the image. The distributions of pattern size are shown in Fig . 4 (b)(c).

To evaluate speed, we had humans judge the pattern size of each image and measured the time taken for each judgment. For accuracy evaluation, we used the human judgment results as the ground truth.

4.2. Result

Fig. 5 provides examples of repeated patterns extracted from fabric. Fig. 6 (a) shows the results of runtime and accuracy, with both increasing as the resolution increases. At a resolution of 300 x 420 pixels, accuracy reaches 100% and runtime is 2.49 seconds. While the runtime is significantly longer than at 100 x 140 pixels, the algorithm still outperforms human processing, which takes 30 seconds per image. Therefore, the algorithm is 12 times faster than human processing, as shown in Fig. 6 (b).

5. Conclusion

We present two approaches for detecting repetition in images. The first approach is based on Fourier analysis, but it is not suitable for our application because it requires additional assumptions about the range of the periods of images. The second approach emulates human processing by shifting and comparing images using the $L_2$ loss function. Despite its time complexity being $O(N^2)$, it outperforms human processing on our dataset.

Reference

Lettry, Perdoch, Vanhoey & Van Gool (2017): Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55.
Park, Brocklehurst, Collins & Liu (2009): Park, M., Brocklehurst, K., Collins, R. & Liu, Y. (2009). Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10). 1804–1816.
Liu & Liu (2013): Liu, J. & Liu, Y. (2013). Grasp recurring patterns from a single view. Proceedings of the IEEE conference on computer vision and pattern recognition. 2003–2010.
Liu, Collins & Tsin (2004): Liu, Y., Collins, R. & Tsin, Y. (2004). A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE transactions on pattern analysis and machine intelligence, 26(3). 354–371.

Computer vision | Alan Hung

Investigation of Algorithms for Retrieving Similar Images

Abstract

1. Introduction

2. Search algorithm

2.1. Per-pixel measures ($L_2$ distance)

2.2. K-Nearest Neighbors (KNN)

2.3. Perceptual similarity (LPIPS)

Feature maps are extracted by passing images $x$ and $x_0$ through a pre-trained network $F$. The distance is computed by subtracting the feature maps and then calculating the spatial average of the $L_2$ norm.

The following shows the spatial average results from ReLU activation layers in AlexNet, based on visually (a) dissimilar and (b) similar input images.

(a)show hide

(b)show hide

3. Experiments

3.1. Dataset

The dataset is used in experiments

3.2. Result

Per-pixel measures ($L_2$) fail to retrieve images with spatial pixel shift, because even small shifts can generate large differences.

Although the color of the retrieved image from KNN may be similar to the input, the structure and context can be different. Therefore, KNN may fail to find similar images due to the lack of spatial information in the features.

Speed (images per second) is displayed on a logarithmic scale to compare different algorithms, with human processing used as the baseline.

4. Application

We have used LPIPS with AlexNet in our in-house library to retrieve similar fabric.

5. Conclusion

Reference

Repeated Pattern Detection of Printed Fabric

Abstract

1. Introduction

Detecting repeated patterns provides insights into objects. (a) Information about the structure and number. (b) Minimum quantity necessary to purchase without having an incomplete pattern.

(a)show hide

(b)show hide

2. Main challenge

3. Program

3.1. Approach with Fourier analysis

Performing FFT on a checkerboard image with different contrasts reveals that in low contrast images, the pattern corresponding to the frequency of maximum power in the spectrum differs from what humans process. (Resolution of image is 250 x 2500 pixels)

(a)show hide

(a)show hide

3.2. Approach with $L_2$ loss

Animation of the algorithm with $L_2$ loss.

4. Experiment

4.1. Dataset

(a) The dataset is used in experiments. Distribution of pattern size in the (b) horizontal and (c) vertical direction.

(a)show hide

(b)show hide

(c)show hide

4.2. Result

Example of repeated patterns extracted in fabric.

(a) Both the accuracy and runtime increase as the resolution increases. (b) At 300 x 420 pixels, the algorithm achieves 100% accuracy and is 12 times faster than human processing. (plot is shown in log scale)

(a)show hide

(b)show hide

5. Conclusion

Reference