Google has worked out how to
transcribe the numbers of houses captured in its Street View
photographs, using a neural network approach.
Recognising arbitrary multi-character text in natural
photographs is a notoriously difficult problem. It's a problem
Google has worked hard to solve since it can add huge amounts of
value to Street View, since it helps to precisely locate each
building within its database. This is especially important in
places where house numbers aren't ordered sequentially as you pass
along the street -- in some places they are numbered based on when
the house was built.
The house numbers captured in Street View are presented in many
different fonts, styles, colours, orientations and character
arrangements, making it highly challenging to identify and
interpret the numbers, particularly when there are additional
environmental factors at play such as lighting and shadows, and
image acquisition issues such as resolution and focus blurs.
Traditional approaches to character recognition involve three
different steps: localisation, segmentation and recognition. Google
engineers -- led by Ian Goodfellow -- propose a unified approach that takes
advantage of a deep convolutional neural network -- with eleven
levels of neurons that have been primed to find numbers.
The researchers relied on
a number of assumptions. The first was that house numbers won't be
longer than five digits, the second is that the building number has
been located and the image has been cropped to make sure that the
number represents at least a third of the frame.
The team used a dataset of around 200,000 street numbers taken
from a publicly available dataset. It focussed on identifying the
entire house number in one go, rather than trying to divide up the
different digits. The aim was to develop a system that could
identify numbers with at least the same accuracy as a human; humans
tend to identify numbers accurately around 98 percent of the
The team managed to achieve the level of accuracy for 95.64
percent of the images, which was deemed good enough to include in
the map databases. The system achieves character level accuracy of
97.84 percent, which improves slightly upon the previous best
system for identifying individual characters (97.54 percent).
The team also tested an internal dataset where Google has
already transcribed tens of millions of street numbers to see how
the system fared. It was a more challenging dataset as it comes
from 12 different countries and has street numbers with non-digit
characters. The team managed an overall sequence transcription
accuracy of 91 percent in this case, but could identify 83 percent
of the numbers with 99 percent accuracy.
The researchers acknowledge that the system only works for
numbers of a fixed length and that the method would be unlikely to
scale well to work for longer strings of numbers.
"Perhaps our most interesting ﬁnding is that neural networks can
learn to perform complicated tasks such as simultaneous
localisation and segmentation of ordered sequences of objects. This
approach of using a single neural network as an entire end-to-end
system could be applicable to other problems, such as general text
transcription or speech recognition," say the authors in their paper
describing the work.