Unlike the new app inspired from HBO’s Silicon Valley, which doesn’t actually have much real world use outside of a few laughs, Yelp’s new hot dog-identifying app works to create a better experience for Yelp users. The neural network identifies over 20 different types of food, categorizing each dish so users can find out what that new diner’s burger looks like without digging through a few hundred photos. Hungry for pizza? Tap the photos, the food option, then click the pizza tab and see if that restaurant you’ve always wanted to try puts big chunks of tomato on top or whether they have deep dish.
“Photos have been trending up through multiple photo apps and the web,” said Goyne, “it’s a new favorite way to consume information. For some of these lower-risk options, like finding an option for lunch where you don’t need to do a bunch of research and you aren’t spending much money, photos are a great way to do that research.”
The software, launching earlier this year, is a result of user feedback. Yelp said that, when it comes to restaurant reviews, some of the most common questions were whether they had a certain dish. By organizing photos uploaded from other users, Yelp is able to present that information in a more precise way.
The convolutional neural network was based on a model built by Google researchers, Miller said. Artificial intelligence platforms are frequently shared among developers so new programs don’t need to start from scratch — like Google’s Wi-Fi-free object recognition algorithm. Yelp gave the system expanded object recognition capabilities by feeding the system thousands of images for each food category. With that data, the computer was able to learn what makes a hot dog different from a sandwich.
Along with the food categories, images are automatically sorted by shots of the interior and exterior and photos of the menu.
Yelp also uses both user response and artificial intelligence to determine which photographs to show first. While the company is continuing to develop software for ordering the photos, the software mixes user feedback — including clicks on “helpful” and “like” — and mixes that with an AI program that puts blurry, dark photos toward the end of the list and images with a nice depth of field and more contrast to the top of the list. The program also tries to prevent similar photos from showing up right next to each other, like three burgers in a row, and determines what image to use as the main photo.
Yelp’s software engineers are also working to teach the computer to recognize what the most important object in the photograph is, so that when the images is cropped to fit on different areas of the web page, the subject is still included.
The software is helping users sift through review photos that, on the most popular restaurants, can number over 1,000. Users won’t see the feature universally, since locations with only a few photographs do not integrate the category organization.
Yelp plans to bring the image classification system into more areas and is currently researching how a similar feature could help users looking for home services and automotive.
