A group of Caltech engineers is teaching computers to recognize trees as if the computer were a plant identification expert by creating visual recognition algorithms. While we can’t clone expert botanists or arborists, we can take tree assessment information and merge it with publicly available LiDAR satellite data and photos from Google Street Level images to create an understanding of images. The effort is being undertaken by the same Cornell/Caltech Visipedia collaborative engineering team that worked with Cornell’s Lab of Ornithology to recognize North American bird species.
“Cities have been surveying their tree populations for decades, but the process is very labor intensive. It usually involves hiring arborists to go out with GPS units to mark the location of each individual tree and identify its species,” says senior author Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering in the Division of Engineering and Applied Science, Caltech. “For this reason, tree surveys are usually only done every 20 to 30 years, and a lot can change in that time.”
The team eventually hopes to develop Visipedia’s capabilities until it can accurately recognize nearly all living things. But they were inspired to turn their attention toward trees when Perona noticed the effects of the California drought on the trees near the Caltech campus in Pasadena.
“I happened to notice that many people in Pasadena were putting drought-resistant plants in their yards to save water, but when they took out the lawns and stopped watering, many trees started dying,” Perona says. “I realized that computer vision might be able to help. By automatically analyzing satellite and street-level images that are routinely collected, maybe we could carry out an inventory of all the trees and we could see over time how Pasadena is changing, whether the trees that are dying are just a few birch trees, which are not native to California and require frequent watering, or whether it’s truly a massive change.”
The engineers say they were also motivated by OpenTreeMap, a collaborative way to crowdsource tree inventory, ecosystem services calculations, urban forestry analysis, and community engagement. They’re going further by building a publicly available and frequently updated tree inventory for cities worldwide.
Although a human could easily look at Google Street Level photographs, spot an object, and ascertain whether or not that object is a tree, the task is not so simple for a computer.
Perona’s research group uses artificial neural networks—algorithms inspired by the brain that allow a computer to “learn” to recognize objects in images. These networks must be trained. “We train an algorithm the way you would teach a child—by showing it lots of examples,” Perona says.
To provide those examples, the team enlisted some human help via a crowdsourcing service called Amazon Mechanical Turk, in which hundreds of workers worldwide can be quickly recruited to complete simple tasks that require human intelligence. In this case, the so-called “turkers” were asked to look at aerial and street-level images of Pasadena and label the trees in each photo. This information was used to train the algorithm to determine which objects were trees.
The engineers next wanted to train the algorithm to identify the species of each tree in the photos—something that the average person cannot do. Fortuitously, the city of Pasadena had partnered in 2013 with Davey Resource Group (DRG) to complete a tree inventory. The survey included species identification, measurements, and the geographical locations of each of the approximately 80,000 trees in the city. Using this information, the engineers trained the algorithm to identify 18 of the more than 200 species of trees in Pasadena.
From Google Maps aerial and street view images, the engineers obtained four different photographs of each tree in Pasadena, taken from different viewpoints and at different distances from the tree. These photos were then analyzed by the algorithm’s “brain”—the artificial neural network. The network then produced a list of a few possible tree species and a score of the certainty of each guess. After comparing the algorithm’s results with those of the 2013 tree survey, the engineers found that their algorithm could detect and identify a tree’s species from Google Maps images with about 80% accuracy.
The computer vision software would continuously collect data about urban street trees from satellite and street level images, which are updated every few months, or from other public images. That information could then be incorporated into software that would help the city understand how its urban forests are evolving.
Cataloging Public Objects Using Aerial and Street-Level Images—Urban Trees The work in Pasadena was supported by the Office of Naval Research, NASA, and Google.
Cataloging Public Objects Using Aerial and Street-Level Images – Urban Trees, IEEE Conference on Computer Vision and Pattern Recognition, by Wegner , J.D., Branson, S., Hall, D., Schindler, K., and Perona, P. in Proceedings IEEE Conference on Computer Vision and Pattern Recognition.
Pl@ntNet is a search engine that indexes plant images based on visual content. It’s organized into databases by location.
LeafSnap, a free mobile app that uses visual recognition to identify tree species from photographs of leaves. Developed by Columbia University, University of Maryland, and the Smithsonian Institution, the app includes trees in the Northeastern US and Canada. Plans are to expand to all trees of the U.S.
NewTerrain October 3, 2016.