I never carry a point-and-shoot camera. Chances are you don’t either. In the last few years, cellphone optics have improved substantially. That means more megapixels, better image sensors and stronger flashes and zooms on the one device most of us carry all the time.
Although image-recognition software is still in its infancy, a number of mobile apps are already translating signs, naming landmarks and providing a running commentary on your world.
Google Goggles, which appeared on Android phones in late 2009 and on the iPhone last year, is best at deciphering landmarks, text, book and DVD covers, artwork, logos, bar codes and wine labels. You start the app — it’s part of Google’s search app for the iPhone — and peer at the object through the camera lens. It takes a stab at identifying it.
I’ve found the app especially useful for comparison shopping. If you’re browsing through a bookstore, for instance, one quick snapshot of a book’s cover allows you to check the price on Amazon. It’s much faster than typing the title into a search bar. Same goes for photographing paintings or craft beer bottles.
Perhaps its most promising use, for tourists especially, is language translation. Goggles can scan English, French, German, Italian, Spanish and, a recent addition, Russian text.
In practice, I’ll admit I’ve had only modest success translating phrases from restaurant menus or street signs. Part of the challenge is capturing an image that’s clear enough for the software to recognize. Unless the text appears on a white background, the software’s success is diminished. But when it does work — wow. Optical character recognition is only going to get better and broader.
Asian languages pose different challenges, says Hartwig Adam, a Goggles engineer. Their alphabets consist of thousands of characters, which tend to be strung together with fewer obvious boundaries. Be wary when buying apps that say they translate Japanese or Chinese. The ones I’ve tried are not fully baked. For now, the handwritten specials posted on the walls of no-frills Chinese restaurants will remain a mystery to me.
Even Google admits Goggles is “not so good” at identifying plants. For that you want Leafsnap, a free iPhone app that supplements a traditional field guide. You photograph a leaf on a white background within the app, which then scans the silhouette. The app then cross-references it with its built-in database. For each potential match, you’re shown high-resolution images of the plant’s leaves, flowers, fruit and bark. Your location is also recorded on a map so you can build a database of your urban forest.
Developed by researchers at Columbia University, the University of Maryland and the Smithsonian Institution, Leafsnap has been downloaded 400,000 times since May. It’s easy to see why. I had a blast trying it out in San Francisco — even though the plants in the app’s database are mostly specific to the New York and Washington areas. In the next 18 months, it will expand to 750 species from 250 species found in the continental United States (excluding South Florida), says Peter Belhumeur, a Leafsnap co-founder and computer science professor at Columbia. Eventually, the app will also use your location to refine its search and improve accuracy, he said.
Goggles has other limitations. It’s not good with faces — deliberately, for privacy reasons. And when I photographed an apple using Goggles on both iPhone and Android handsets, no close matches were found. Minutes later, I tried Meal Snap, a $2.99 iPhone app meant as a tool for dieters. Not only did it correctly identify the fruit, but Meal Snap also provided an accurate caloric range (60 to 90 calories).
Even more impressive was what happened when I used the app to deconstruct a bowl of homemade chopped salad. The app correctly identified diced beets and sliced cherry tomatoes alongside broccoli in the tossed mess. That said, the app failed to spot my spinach, deli turkey, cauliflower and bits of pepperoncini. However, the calorie estimate was only off by about 100 calories; and it was actually inflated, which is O.K. if your goal is weight loss.
Another photo-recognition tool with health implications is Skin of Mine, a $2.99 iPhone app that analyzes moles and freckles.
Start the app, and an outline of a human body appears. Once you touch a spot on the body, the camera opens. After taking a snapshot, you trace a mole using a tiny on-screen pencil. From there, the mole’s symmetry, border and color are analyzed and assigned numerical values. By cross-checking the numbers with a database of a few hundred images culled from dermatologists, the app estimates whether your mole might be consistent with melanoma.
“The idea isn’t to replace your dermatologist, but keep an eye on things between doctor visits,” says Ellen Kislal, president of Medical Image Mining Laboratories, which developed Skin of Mine. For users in Connecticut, Louisiana, Maryland, Nebraska, New Jersey and New York, the app also provides the option of connecting online with a doctor. For $40 to $65, the doctor will examine your photo and make a diagnosis. Eventually, the app will expand to include assessments of other skin conditions like acne, vitiligo and wrinkles, Dr. Kislal said.
When it comes to comparison shopping for books, CDs or DVDs, an alternative to Goggles is SnapTell, which is available for Android and Apple phones. Like Goggles, the free app features a bar code scanner. Still, I find photographing items much more satisfying. Once you snap a picture, the app determines online prices at sites like Amazon, Alibris and Textbooks.com (SnapTell was acquired by Amazon in 2009). Better still, the app uses your location to call up prices at brick-and-mortar retailers. Another location-based app is Snooth Wine Pro, which is intended for wine lovers. Take a close-up of a bottle’s label, and the $4.99 iPhone app not only displays the price, but also maps nearby bodegas, liquor stores and wine cellars. I photographed six bottles of wine, including a cheap red, a more expensive pinot and sake. The app recognized five of the six. Other bells and whistles for hardcore oenophiles include user-submitted reviews, a wish list and private tasting notes. Even if you are not a connoisseur, don’t settle for the free version of the app; it doesn’t include photo recognition.
Beyond browsing books or window-shopping wines, you can also buy fine art using photo ID. ArtMatch, a free iPhone app developed by Art.com, the online poster and print company, has a database of more than a million works. It’s ideal for museums, restaurants or doctor’s offices, basically anywhere where you’re likely to see well-known originals or prints. I found artMatch effective at spotting a Van Gogh self-portrait, but not art from contemporaries like Geoff McFetridge or Rachell Sumpter.
It is difficult to predict how these apps will evolve, but one thing is certain: they are already changing the way we see our world.
-Steven Leckart, The New York Times
