Volume 2, Issue 8 October 2002

2002
2001

Browsing Art Collections, Bit by Bit by David Pescovitz This top-level view of the Clustering Art interface shows a representative image from each of the clusters. (Click for larger image.) Browsing a large museum of fine art can be an overwhelming experience. On the other hand, losing yourself in the galleries can be rewarding. But if you're seeking a particular subject matter – say, representations of horses – browsing is like searching for a needle in a haystack. And to make matters worse, the difficulty increases exponentially if you're not certain whether the collection you are browsing even contains a single image of a horse. To help navigate the sprawling art landscape, UC Berkeley Computer Science professor David Forsyth and his graduate students are integrating computer vision technology with natural language processing to create visual summaries of massive art collections available online. "In some sense, a search is meaningless unless you know what you're going to find," Forsyth says. The researchers recently demonstrated their clustering art method using 10,000 images of line drawings, paintings, sculpture, and ceramics from the Fine Arts Museum of San Francisco. The project is part of the digital library research efforts within the Center for Information Technology Research in the Interest of Society (CITRIS). At the heart of Forsyth's method is computer vision technology developed at Berkeley. The computer vision system segments the images into sections for computer analysis. As Forsyth demonstrates, images containing large vertical features with similar colors against a light-colored background, for instance, are grouped together. After the visual information is gathered, the software scours the "meta-data," captions for each image containing short physical descriptions, subjective interpretations, or mood characterizations previously entered by museum volunteers. Clicking on the venetian glass image in the top-level view (photo at top) reveals the elements in that particular cluster. (Click for larger image.) Courtesy David Forsyth "The next step is for the computer to probabilistically cluster together pictures that tend to have regions and words that are the same," Forsyth says. "Then for each cluster, it pulls out the picture that's most like the others." As if by magic, clicking on an image of a female figurine on a screen reveals almost a dozen other statues of females along with several illustrations of visually-similar subjects. Other clusters – fruit or painted plates, for instance – also make thematic sense even with one or two inconsistencies. According to Forsyth, "It is a remarkable fact that, while text and images are separately ambiguous, jointly they tend not to be. This is probably because the writers of text descriptions of images tend to leave out what is visually obvious (the colors of flowers, etc.) and mention properties that are very difficult to infer using vision (the species of the flower, say). Can Clustering Art change the way we interact with museums and other image collections? We want to hear from you... Clicking on any of the images within a cluster links to additional information about the work, as provided by the museum. By clustering the artworks using the complementary image and text information, Forsyth and his team have summarized the 8,000 images in their sample with just 80 representative images. "If you're a small museum or not a famous one, very few people will know what's in your collection," Forsyth says. "But this way they can get a sense by browsing your online collection so they can determine if it's worth a detour to pay the actual museum a visit." The researchers are also exploring the clustering method to "auto-illustrate" text. For example, querying the image database with a passage from Moby Dick yields 20 illustrations of period ship drawings, several containing large whales. Forsyth's future research plan is to explore clustering news. The endless glut of news articles and photojournalism available online, he says, is data that could be probabilistically clustered to gain insight into public consciousness. "Wouldn't it be nice if you could collect every news article with pictures in one place and organize them visually to get a sense of the distribution of opinion on a particular topic." Forsyth says. Clustering Art David Forsyth's Home Page CITRIS Lab Notes is published online by the Public Affairs Office of the UC Berkeley College of Engineering. The Lab Notes mission is to illuminate groundbreaking research underway today at the College of Engineering that will dramatically change our lives tomorrow. Editor, Director of Public Affairs: Teresa Moore Writer, Researcher: David Pescovitz Designer: Robyn Altman Subscribe or send comments to the Engineering Public Affairs Office: lab-notes@coe.berkeley.edu. © 2002 UC Regents. Updated 9/30/02.