Algorithm | Alan Hung

Investigation of Algorithms for Retrieving Similar Images

Tue, 10 Jul 2018 00:00:00 +0000

Table of Contents

Abstract

Automating the task of searching for similar images of fabric is crucial to improve efficiency in the fabric-related industry. Three algorithms were investigated in this project: per-pixel measures ($L_2$), K-Nearest Neighbors (KNN), and perceptual similarity (LPIPS). While LPIPS provides a better way to retrieve similar fabric as perceived by humans, its computational expense becomes a main obstacle when the number of images increases.

1. Introduction

Searching for similar images of fabric is a common task in the fabric-related industry. For instance, buyers may want to purchase fabrics similar to previous best-selling ones, while designers may want to experiment with different yet similar fabrics during the product design process.

Although humans can recognize similar images effortlessly, traditionally, this task has relied on human search through all available fabrics to find a similar one, which is time-consuming and exhausting. Automating this procedure is crucial to improve efficiency.

2. Search algorithm

Since we are searching through all images, we will use the notation $N$ to represent the total number of images in the following discussion.

2.1. Per-pixel measures ($L_2$ distance)

One of the classic methods for comparing images is to calculate the difference of every pixel using the $L_2$ distance. The time complexity is $O(N^2 * C * W * H)$, where $C$ represents the number of channels in the image, and $W$ and $H$ represent the width and height of the image, respectively. The space complexity is $O(N^2)$ because we need to store all the distances between the images.

2.2. K-Nearest Neighbors (KNN)

Rather than comparing the images directly, we can extract features from the image, such as the main color, color schemes, pattern size, and so on. Similarity can be defined as the $L_2$ distance between features. However, as the number of features increases, all samples may appear dissimilar, which is often referred to as the “curse of dimensionality”. To overcome this problem, we can reduce the dimensionality using PCA.

The time complexity is $O(N^2 * K)$, where $K$ represents the number of principal components from PCA. The space complexity remains $O(N^2)$ since we store all distances.

2.3. Perceptual similarity (LPIPS)

This method, which was published by Zhang ( Citation: Zhang, Isola & al., 2018 Zhang, R., Isola, P., Efros, A., Shechtman, E. & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595. ) as shown in the Fig. 1, extracts feature maps from activation layers and calculates the weighted distance $d_0$ between those feature maps. To apply this method, we can push images $x$ and $x_0$ through a pre-trained network (such as AlexNet ( Citation: Krizhevsky, 2014 Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997. ) ), denoted as $F$.

Fig. 2 provides an example with AlexNet. When the input images exhibit visual differences, as in Fig. 2 (a), the spatial average result differs significantly from that of the similar input images in Figure Fig. 2 (b).

While this method is proven to be effective in finding similar images, it is computationally expensive. Depending on the architecture of the pre-trained network, which may involve multiple convolutional and pooling layers, the runtime may be significantly increased. In addition, we will need additional memory to store all feature maps from activation layers.

3. Experiments

3.1. Dataset

We collected 113 digital images of fabric with dimensions of 30 cm x 42 cm (3500 x 4200 pixels), as shown in Fig. 3. To test the algorithm’s ability to retrieve similar images, we randomly select images from the dataset, add spatial pixel shifts, and then put them back into the dataset. We then test whether the algorithm can retrieve them.

3.2. Result

As shown in the Fig. 4, the use of per-pixel measures ($L_2$) may fail to retrieve images with spatial pixel shift. This is because even a small shift can result in a large difference. In contrast, KNN and LPIPS rely solely on extracted features and can therefore successfully find those images.

However, due to the lack of spatial information in features, KNN might still fail to find images that are similar from a human perspective. As shown in the Fig. 5, the KNN algorithm retrieves images that are similar to the input image only in terms of colors. However, the context and structure of these images are entirely different.

As previously discussed, in terms of runtime (as shown in the Fig. 6), LPIPS is the most computationally expensive of the three algorithms, taking up to 8.3 seconds per image on our dataset.

4. Application

To improve the efficiency of fabric searching, LPIPS with AlexNet has been implemented into the in-house library, as shown in the Fig. 7. When a new image is imported, the algorithm retrieves similar fabrics and stores them in a database.

5. Conclusion

Three algorithms were investigated in this project. The per-pixel measure ($L_2$) did not meet our requirements due to its sensitivity to pixel shift. Similarly, KNN is not applicable since the features lack spatial information. LPIPS provides a better way to retrieve similar fabric as perceived by humans, but its computational expense becomes a main obstacle when the number of images increases.

Reference

Zhang, Isola, Efros, Shechtman & Wang (2018): Zhang, R., Isola, P., Efros, A., Shechtman, E. & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595.
Krizhevsky (2014): Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997.

Repeated Pattern Detection of Printed Fabric

Thu, 01 Feb 2018 00:00:00 +0000

Table of Contents

Abstract

We study the problem of detecting repeated patterns. Our goal is to extract a single unit of repeated patterns from an input image. This problem can be simplified for printed fabric because of its strictly regular and fronto-parallel nature. Under these properties, we investigate two algorithms. The first one uses Fourier analysis, but it is not suitable for our purpose because it requires additional assumptions about the period of the image. The second algorithm involves shifting the image and calculating the distance using the $L_2$ loss function. This method can efficiently solve the problem compared to human processing on our dataset.

1. Introduction

Repeated patterns are prevalent, particularly in human-made objects, as illustrated in the Fig. 1 (a). They offer insights into the object, such as allowing us to estimate the number of floors and the structure of a building ( Citation: Lettry, Perdoch & al., 2017 Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55. ) or count the number of objects in an image ( Citation: Park, Brocklehurst & al., 2009 Park, M., Brocklehurst, K., Collins, R. & Liu, Y. (2009). Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10). 1804–1816. ) .

This is especially important for printed fabric in the textile industry, as it consists of repeated patterns in both the vertical and horizontal directions. Extraction a single pattern is useful in two ways. First, It provides buyers with the minimum quantity necessary to purchase without having an incomplete pattern. For example, one could consider purchasing two units in the vertical direction to have two birds in opposite direction in a single pattern, as shown in the Fig. 1 (b). Second, organizing a library by patterns can help designers find patterns more efficiently, ultimately speeding up the design process.

2. Main challenge

Automating repetition detection remains a challenging task since the definition is unclear. For example, a checkerboard consisting of black and white has perfect repetition. However, it is often accompanied by irregularities, as shown in the Fig. 1 (a), the windows of the building are similar to each other, rather than being exactly the same in each pattern.

Several algorithms have been developed to address this deviation ( Citation: Liu, Collins & al., 2004 Liu, Y., Collins, R. & Tsin, Y. (2004). A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE transactions on pattern analysis and machine intelligence, 26(3). 354–371. ; Citation: Liu & Liu, 2013 Liu, J. & Liu, Y. (2013). Grasp recurring patterns from a single view. Proceedings of the IEEE conference on computer vision and pattern recognition. 2003–2010. ) , while others make assumptions with a nearly regular pattern ( Citation: Lettry, Perdoch & al., 2017 Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55. ) . In other words, assumptions are crucial for implementing an algorithm.

3. Program

Although it is a complex problem, fortunately, for our purposes, it is limited to printed fabric, which is both strictly regular and fronto-parallel due to its manufacturing process. As a result, our problem is as simple as detecting the perfect repetition of a checkerboard. The problem’s constraint depends on the resolution of the image, which in our case is 3500 x 4200 pixels.

Constraints:

len(pixels) = 3500 x 4200

In the following analysis, we will use the notation $N$ to represent the total number of pixels in an image.

3.1. Approach with Fourier analysis

The intuitive method for identifying the period would be to use Fourier analysis, which is made efficient by using FFT. This results in a time complexity of $O(NlogN)$. However, since the basic idea behind Fourier analysis is an approximation by sums of simpler trigonometric functions, identifying the peak in spectrum can be challenging.

For instance, let’s say we perform Fourier analysis in the horizontal direction of a checkerboard. As shown in the Fig. 2 (a), the frequency of the maximum power corresponds to the pattern consisting of a pair of black and white squares. If we reduce the contrast, as also depicted in the Fig. 2 (b), humans can still effortlessly identify the pattern. However, the position of the maximum power in the spectrum shifts to a lower frequency. In other words, to find the same pattern as that perceived by humans, we need to specify the search range in the spectrum. Nevertheless, since there is no certain range of periodicity for images, this approach is not practical.

3.2. Approach with $L_2$ loss

This approach mimics human processing by sliding images to find their periodicity. In other words, it compares the pixels of images one by one, and uses the $L_2$ loss function as a metric for the distance between images. Here is how the algorithm works, as shown in the Fig. 3:

Create a copy of the original image.
Fixed one image as a reference and slide another.
- Calculate the loss in each RGB channel.
- Calculate the average loss for overlapped regions.
Calculate the midpoint of the average loss function.
Measure the distance between two midpoints while ascending or descending.

The size of the pattern can be extracted by performing the algorithm in both horizontal and vertical directions. The algorithm’s time complexity is $O(N^2)$ since we need to iterate through all pixels in each iteration. The space complexity is $O(N)$ because we create a copy of the original image.

Although the algorithm is $N^2$, all elements share the same computation at every step. Therefore, we can take advantage of vectorization by using Numpy. Additionally, it is necessary to resize the image for the algorithm, which involves a trade-off between runtime and accuracy.

4. Experiment

4.1. Dataset

To test the speed and accuracy of our algorithm, we collected 113 digital images of fabric with dimensions of 30 cm x 42 cm (3500 x 4200 pixels), as shown in Fig. 4 (a). The pattern size was extracted manually using the same method as the algorithm, which involved shifting and comparing the image. The distributions of pattern size are shown in Fig . 4 (b)(c).

To evaluate speed, we had humans judge the pattern size of each image and measured the time taken for each judgment. For accuracy evaluation, we used the human judgment results as the ground truth.

4.2. Result

Fig. 5 provides examples of repeated patterns extracted from fabric. Fig. 6 (a) shows the results of runtime and accuracy, with both increasing as the resolution increases. At a resolution of 300 x 420 pixels, accuracy reaches 100% and runtime is 2.49 seconds. While the runtime is significantly longer than at 100 x 140 pixels, the algorithm still outperforms human processing, which takes 30 seconds per image. Therefore, the algorithm is 12 times faster than human processing, as shown in Fig. 6 (b).

5. Conclusion

We present two approaches for detecting repetition in images. The first approach is based on Fourier analysis, but it is not suitable for our application because it requires additional assumptions about the range of the periods of images. The second approach emulates human processing by shifting and comparing images using the $L_2$ loss function. Despite its time complexity being $O(N^2)$, it outperforms human processing on our dataset.

Reference

Lettry, Perdoch, Vanhoey & Van Gool (2017): Lettry, L., Perdoch, M., Vanhoey, K. & Van Gool, L. (2017). Repeated pattern detection using CNN activations. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 47–55.
Park, Brocklehurst, Collins & Liu (2009): Park, M., Brocklehurst, K., Collins, R. & Liu, Y. (2009). Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10). 1804–1816.
Liu & Liu (2013): Liu, J. & Liu, Y. (2013). Grasp recurring patterns from a single view. Proceedings of the IEEE conference on computer vision and pattern recognition. 2003–2010.
Liu, Collins & Tsin (2004): Liu, Y., Collins, R. & Tsin, Y. (2004). A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE transactions on pattern analysis and machine intelligence, 26(3). 354–371.

Color Theme Extraction Based on the Bucket Sort Algorithm

Wed, 05 Jul 2017 00:00:00 +0000

Table of Contents

Abstract

We present an algorithm for extracting color themes from digital images, which is crucial for automating the design process. The algorithm uses the HSV color space and a bucket sort with binary search to achieve up to 98% accuracy on a dataset of 113 images with a runtime of 157 milliseconds per image, representing a 63x gain in runtime compared to human processing.

1. Introduction

Color is a primary factor in product design and significantly impacts various areas, including sales, marketing, and customer emotions ( Citation: Sigh, 2006 Sigh, S. (2006). Impact of color on marketing. Management decision, 44(6). 783–789. ) . While it may seem like a color palette is subjective and only related to personal preference, designers actually rely heavily on color theory to make their decisions.

Modern color theory is largely based on Isaac Newton’s color wheel ( Citation: Newton, 1730 Newton, I. (1730). Opticks: Or, A Treatise of the Reflections, Refractions, Inflections and Colours of Light. William Innys at the West-End of St.Paul's. ) , which illustrates the relationship between colors. Unlike RGB, the color wheel is in the HSV color space, which is essentially a cylindrical coordinate system where hue, saturation, and value are related to degree, radius, and height, respectively. The cross-section of the cylinder represents the color wheel.

Color themes are combinations of colors on the color wheel. For example, as shown in Fig. 1, an analogous color theme can be created by selecting adjacent colors, while a complementary color theme consists of two colors that are directly across from each other.

2. Main challenge

When designing a product composed of different fabrics, the designer relies on color themes of each fabric, as shown in Fig. 2, which are manually extracted by other workers. This process is time-consuming and costly. To expedite the design process, it is crucial to find a way to extract color themes automatically.

Although the theory behind color themes is mature and there are commercial tools available, such as Adobe Color Wheel and Pantone Connect, that can generate color themes or extract colors from imported images, these tools are still unable to identify color themes directly from images and do not provide an API. As a result, automating the design process with these tools is impossible.

3. Program

To implement our own algorithm, the first step is to analyze the problem. This can be broken down into three parts:

Converting the RGB channels of the digital image into HSV format.
Defining color themes quantitatively.
Implementing a search algorithm to check if colors match the defined themes.

3.1. Color Conversion

While it is possible to implement the conversion algorithm based on recent work ( Citation: Chernov, 2015 Chernov, V. (2015). Integer-based accurate conversion between RGB and HSV color spaces. Computers & Electrical Engineering, 46. 328–337. ) , which claims to be faster and more accurate than classical methods. Conversion using an image processing package like OpenCV is already fast enough for our purposes.

3.2. Quantitative definition of a theme

Since visual artwork rarely consists of only a single color, it is more appropriate to use intervals to describe a range of colors. By considering the relative position of each interval, we can define a theme. As a result, a color theme can be described using three parameters:

The number of intervals
The range of each interval
The relative position of each interval

For example, in Fig. 1, the triadic theme consists of three intervals, each with a relative position of 120 degrees and a range that can start from 15 to 100.

3.3. Search algorithm

Before delving into the algorithm, it is important to consider the constraints of the problem. According to the HSV color space, colors are confined to a range between 0 and 359. Additionally, in our dataset, we resize all images to 100 x 140 pixels, striking a balance between runtime and accuracy. Taking monochrome, analogous, complementary, and triadic themes into account, there are 22,680 possible themes when considering all combinations of numbers and ranges in intervals.

Constraints:

0 ≤ color ≤ 359
1 ≤ len(color) ≤ 14000
len(theme) = 22680

In the following analysis, we use the notations $N$ and $K$ to represent the lengths of the “colors” and “themes” lists, respectively.

3.3.1. Naive approach

A naive approach would be to iterate through all colors and theme until finding the correct one. Here’s how the algorithm works:

Iterate through all themes.
- At each iteration, we iterate through all colors.
- Check if theme covers all colors.
  - If it does, return it.
  - Otherwise, we move on to the next one.

The time complexity is $O(NK)$. Although we need extra memory to store the number of colors in each interval during each iteration, the space complexity remains $O(1)$ since it only depends on the number of intervals in each theme.

3.3.2. Optimized approach using Bucket sort

Since there are upper and lower bound of color, the problem can be optimized using a bucket sort with binary search.

Here’s the basic idea of the algorithm:

To divide the range of colors (0~360) into “buckets”, select a bucket size. For example, if the bucket size is 10, there will be 36 buckets in total.
Iterate through each color and use binary search to find the correct bucket.
Merge adjacent and non-empty buckets.
Iterate through all themes.
- At each iteration, iterate through all merged buckets.
- Check if the theme covers all buckets.
  - If it does, return it.
  - Otherwise, move on to the next one.

Assuming that $B$ represents the number of buckets, the time complexity of this algorithm is $O(NlogB + BK)$, where $NlogB$ comes from bucket sort with binary search. The space complexity is $O(B)$. The bucket size is considered a hyperparameter. In practice, a bucket size of 5 or 10 is sufficient to obtain accurate results. As a result, this algorithm significantly improves speed compared to a naive approach.

4. Experiment

4.1. Dataset

To test the speed and accuracy of our algorithm, we collected 113 digital images of fabric with dimensions of 3500 x 4900 pixels, as shown in Fig. 4. For the algorithm, we resized the images to 100 x 140 pixels and used a bucket size of 10.

To evaluate speed, we had humans judge the theme of each image using a commercial color picker and measured the time taken for each judgment. For accuracy evaluation, we used the human judgment results as the ground truth, as shown in Fig. 5.

4.2. Result

The algorithm only failed on 2 images, resulting in an accuracy of 98%. The reason for these failures is that the images are composed of multiple color themes. For example, the image in Fig. 6 is considered to be both complementary and analogous.

The comparison of runtime is shown in Fig. 7. Excluding the time for importing images, the average runtime for processing a single image is 157 milliseconds for the algorithm and 10 seconds for a human. This represents a 63x gain in runtime compared to human processing. For reference, a naive approach could take up to 30 seconds, which is even slower than human processing.

5. Conclusion

We have presented an algorithm for identifying color themes that does not require any training or human annotations. The algorithm is 63 times faster than human processing and achieves up to 98% accuracy on our dataset.

Reference

Sigh (2006): Sigh, S. (2006). Impact of color on marketing. Management decision, 44(6). 783–789.
Newton (1730): Newton, I. (1730). Opticks: Or, A Treatise of the Reflections, Refractions, Inflections and Colours of Light. William Innys at the West-End of St.Paul's.
Chernov (2015): Chernov, V. (2015). Integer-based accurate conversion between RGB and HSV color spaces. Computers & Electrical Engineering, 46. 328–337.

Algorithm | Alan Hung

Investigation of Algorithms for Retrieving Similar Images

Abstract

1. Introduction

2. Search algorithm

2.1. Per-pixel measures ($L_2$ distance)

2.2. K-Nearest Neighbors (KNN)

2.3. Perceptual similarity (LPIPS)

Feature maps are extracted by passing images $x$ and $x_0$ through a pre-trained network $F$. The distance is computed by subtracting the feature maps and then calculating the spatial average of the $L_2$ norm.

The following shows the spatial average results from ReLU activation layers in AlexNet, based on visually (a) dissimilar and (b) similar input images.

(a)show hide

(b)show hide

3. Experiments

3.1. Dataset

The dataset is used in experiments

3.2. Result

Per-pixel measures ($L_2$) fail to retrieve images with spatial pixel shift, because even small shifts can generate large differences.

Although the color of the retrieved image from KNN may be similar to the input, the structure and context can be different. Therefore, KNN may fail to find similar images due to the lack of spatial information in the features.

Speed (images per second) is displayed on a logarithmic scale to compare different algorithms, with human processing used as the baseline.

4. Application

We have used LPIPS with AlexNet in our in-house library to retrieve similar fabric.

5. Conclusion

Reference

Repeated Pattern Detection of Printed Fabric

Abstract

1. Introduction

Detecting repeated patterns provides insights into objects. (a) Information about the structure and number. (b) Minimum quantity necessary to purchase without having an incomplete pattern.

(a)show hide

(b)show hide

2. Main challenge

3. Program

3.1. Approach with Fourier analysis

Performing FFT on a checkerboard image with different contrasts reveals that in low contrast images, the pattern corresponding to the frequency of maximum power in the spectrum differs from what humans process. (Resolution of image is 250 x 2500 pixels)

(a)show hide

(a)show hide

3.2. Approach with $L_2$ loss

Animation of the algorithm with $L_2$ loss.

4. Experiment

4.1. Dataset

(a) The dataset is used in experiments. Distribution of pattern size in the (b) horizontal and (c) vertical direction.

(a)show hide

(b)show hide

(c)show hide

4.2. Result

Example of repeated patterns extracted in fabric.

(a) Both the accuracy and runtime increase as the resolution increases. (b) At 300 x 420 pixels, the algorithm achieves 100% accuracy and is 12 times faster than human processing. (plot is shown in log scale)

(a)show hide

(b)show hide

5. Conclusion

Reference

Color Theme Extraction Based on the Bucket Sort Algorithm

Abstract

1. Introduction

Relative position of colors for different themes.

Analogousshow hide

Complementaryshow hide

Triadicshow hide

2. Main challenge

When designing a product, the designer should consider the color theme of the fabric. In this example, the complementary colors are dark green and pink. Therefore, the strap, zipper, and other fabric of the product should also match this color theme.

3. Program

3.1. Color Conversion

3.2. Quantitative definition of a theme

3.3. Search algorithm

3.3.1. Naive approach

3.3.2. Optimized approach using Bucket sort

Animation of various search algorithms.

(a) Naive approachshow hide

(b) Optimized approach with bucket sortshow hide

4. Experiment

4.1. Dataset

The dataset is used in experiments involving monochrome, analogous, complementary, and triadic color themes.

To evaluate accuracy, we use human judgement as the ground truth. For instance, the algorithm extracted analogous, complementary, and triadic color themes from the following images, which match the result obtained from human judgement using a color picker.

4.2. Result

The algorithm fails to extract if the image is composed of multiple color themes.

By using an optimized approach in the search algorithm, the processing speed is 63 times faster than human processing. The unit of measurement for processing speed is images per second, and the plot is shown on a log scale.

5. Conclusion

Reference