Xian-Sheng Hua – Everyday is a nice day

On June 3, Microsoft released a new version of its Bing search engine. Among the new functionalities, an interactive image search interface provides a totally new capacity of searching images on the Web. This article will introduce this feature with a couple of interesting examples.

The new feature is based on a research effort from Microsoft Research Asia (MSRA). Last year, the two major contributors for this technique from MSRA, Xian-Sheng Hua and Jingdong Wang, has reported this technology using a number of screenshots of a prototype in MSRA Blog (Chinese version). Now this interesting technique is in Bing image search that everyone can try.

Originally the feature is called “search by color map” in the prototype. But now it is called “Specific color” and “Filter using sketch” in Bing image search. We can call it “image search by color sketch”.

In this article, we will first introduce the steps of using this feature, which has three functionalities, and then present more details and tips of using this feature. And last we will talk a little bit about the technology behind.

Table of Content

How to Use Search by Color Sketch

More Examples

Grass 
Flower 
Flag 
Search people (“Lady Gaga”) 
Search car (“BMW”)

About the Underlying Technologies

How to Use Search by Color Sketch

1. Visit http://www.bing.com and click “Images”, or directly visit: http://www.bing.com/images or http://image.bing.com;

2. Check the top-right corner of the web page and change the region of the service to “United States” if currently it is not. Currently this feature is only enabled in US region, but can be used worldwide as long as you changed the region to “United States”;

3. Type a query in the search box and press search button. Now we use “apple” as an example.

4. Click “COLOR” on the left panel (if it is not expanded), then you will see a menu “Specific color”. This is the entrance of the new feature. Currently it is hidden and hope it will be placed somewhere easier to be found (Figure 1).

(a) The whole page

(b) Enlarged: the bottom-right corner of the page

Figure 1: Open the entrance of the new feature.

5. After clicking, you get a panel at bottom-left part of the page (Figure 1), where you can try the first functionality: search by dominant color. That means, you can get “flower” related images but the dominant color of the images would be the one you clicked. For example, if you click the small blue block, you will get results like Figure 2 (mostly blue flowers or flowers’ color close to blue) (Figure 2)

Figure 2: Blue flowers (“zoom small” was clicked so much more images can be shown in one page, which can be set on top-right of the image area)

6. The second and third functionality needs open the drawing panel by checking “Filter using sketch” in the color panel. Then you can choose colors and draw the color sketch of the desired image (Figure 3). Please note this is not for drawing contour of a cat, dog, house or other objects, but only the color “map” of desired images. For example, we may want to find purple flowers with green leaf – that’s what Figure 3 is showing. After drawing the color sketch, press “Apply” to get new search results. We can call this “search by drawing” or “search by color sketch”.

Figure 3: Purple flowers with green leaves.

7. The third functionality could be called “search by modified example”. That means, instead of drawing on the empty panel, we can drag-n-drop an image from the search results to the panel. We can directly click “Apply” to search similar images (Figure 4), or we can draw something to change the example before search. For example, we use the eraser to ease the green leaves and then draw some yellow blocks, which indicates that I’d like to see some yellow followers or parts besides the purple ones (Figure 5).

Figure 4: Search by an example (drag-n-drop an image from the search results).

Figure 5: Search by modified example (purple flowers but with something yellow). The first image in the results will remain the same as the originally selected image no matter how you modified it. This is by design, so people will always know which image he or she has chosen.

There are hyperlinks embedded in the example pictures above (and also those in next section). You can click it to see what it looks like in Bing image search. However, the results may be slightly different (or even much different if you read this article long after June 2010) with the pictures showing here as Bing’s image index is keeping updating. If the links are invalid, you may try it manually using the same term and similar drawing, or any other drawings you like.

Below figure shows the functions of the button in the panel (Figure 6).

Figure 6: Functions of the buttons.

More Examples

We will show you a number of examples, based on which you will learn how to well express you intent through drawing color sketch.

Grass

Type “grass” in Bing image search you will get something like Figure 7.

Figure 7: Original results of “grass” in Bing image search.

If want more sky on the top and less grass at the bottom, you can draw something to express this intent and get something like Figure 8.

Figure 8: “Grass” images with more sky on top and less grass at bottom.

Or you want less sky on top and more grass at bottom (Figure 9):

Figure 9: “Grass” images with little sky and lots of grass.

Sometimes you may want to find “orange grass” (Figure 10), or you find one of the orange grass looks more like what you want (drag it to the panel as an example) (figure 11).

Figure 10: Orange grass.

Figure 11: Search by example “grass”

Sometimes, you may want to get some grass images with clean background, so it will be easier to use it. This can be achieved by set the background to white or black or any other possible background color (Figure 12).

Figure 12: Grass images with clean/white background

Another example of searching by example: grass with orange sky (Figure 13), and modified example (similar grass with blue sky) (Figure 14).

Figure 13: Search by example (grass with orange sky area).

Figure 14: Search by modified example (similar grass but with blue sky). Again, the first image in the results will remain the same as the originally selected image no matter how you modified it. This is by design, so people will always know which image he or she has chosen.

Flower

Original results of searching “flower” in Bing image search (Figure 15). Blue flowers or some flower related examples have been shown in Figure 2, 3, 4 and 5. Here are more examples: purple flower (Figure 16), yellow flower (Figure 17), yellow flower with dark background (Figure 18), search by example (an yellow flower with dark background) (Figure 19), and search by modified example (yellow flower with green background)(Figure 20).

Figure 15: Original results of “flower” in Bing image search.

Figure 16: Purple flowers (by selecting the dominant color).

Figure 17: Yellow flowers (indicated by yellow blocks in the center area).

Figure 18: Search by drawing (yellow flowers with dark background, indicated by yellow blocks at center area and black blocks at the border).

Figure 19: Search by example (yellow flower with dark background).

Figure 20: Search by modified example (change the background to green, indicating leaves or grass).

Flag

When you saw a flag of a country when watching world cup, but you don’t know what the country of the flag, you may use Bing image search to find it out. Below shows several results (Figure 21 to Figure 27) – after drawing the flag roughly, you get the real flag images. Mouse-over or click the desired result, you will know the country of the flag.

Figure 21: China flag.

Figure 22: India flag (and other similar flags, such as Miami city flag and Niger flag)

Figure 23: Niger flag (and other similar ones).

Figure 24: Ghana flag.

Figure 25: Mauritius flag (search by example).

Figure 26: Gambia flag (search by modified example).

Figure 27: Danmark flag and similar flags (search by drawing).

My son (2.5 years old) is able to recognize more than 80 flags of different countries, but I can only recognize 10+. Now with Bing’s “search by color sketch”, I can compete with him now, thought the speed is much slower than him 🙂

Search people (“Lady Gaga”)

“Lady Gaga” is very popular recently. Figure 28 shows the initial results in Bing. Figure 29, 30, 31, and 32 shows the images of Lady Gata walking on red carpet and wearing in different colors.

Figure 28: Original results of “Lady Gaga” from Bing image search.

Figure 29: Lady Gaga walking on red carpet (indicated by a red line at the bottom).

Figure 30: Lady Gaga walking on red carpet and wearing in black (indicated by a red line at the bottom and some black blocks near the center.

Figure 31: The same as above but wearing in white (sometimes you need to adjust the size and/or position of the color blocks to get better results).

Figure 32: Walking on red carpet while background is white.

Figure 33: Search by an example (wearing in black and blue).

Figure 34: Dark background.

Figure 35: Dark background with larger face area.

Figure 36: Search by another example with dark background.

Search car (“BMW”)

Figure 37 to Figure 43 show a couple examples for search color of different colors and different background.

Figure 37: Original results of “BMW” of Bing.

Figure 38: Blue BMW.

Figure 39: Red BMW.

Figure 40: Red BMW on lawn.

Figure 41: Red BWM on lawn with blue sky.

Figure 42: Search by modified example. Originally the car is black (i.e., the first image in the result list). Now it was changed to red – we got red cars in similar background.

Figure 43: The same as the above example, but we change the color of the car to orange.

About the Underlying Technologies

This is not re-ranking, not conventional “query by sketch”, not just user-interface. It should be the first web-scale “image search by drawing” system. When drawing is provided, a new query based on the text query and drawing will be issued to search new images from the entire Bing index.

……

(To be continued – some high-level introduction of the technology and the differences compared with existing work, as well as some tips of using the feature.)

By Xian-Sheng Hua, Beijing, June 2010