Bees, beetles and moths have compound eyes with multiple lenses that together form a single image on an underlying array of light-sensitive photoreceptor cells. In theory these small lenses should be far worse at night vision than the larger human eye, but the way the cells in an insect's visual system process the light signals means they can make much better use of what little light is available. As light levels drop, the neuronal wiring in the insect's eye can either pool the signals from neighbouring photoreceptors or collect the signals for longer periods, Warrant says. Neither strategy is perfect: summing in time makes it difficult to detect movement while summing in space throws away spatial detail. It seems, though, that an insect's eyes can make a trade-off between the two strategies depending on how much light is available and how quickly the insect is moving relative to its surroundings.
In 1999, Warrant developed a mathematical model to describe the optimal "spatiotemporal summation" strategy for any given light intensity and speed of image movement (Vision Research, vol 39, p1611). However, he now believes that insect eyes are even more sophisticated than this. His early model calculated just one level of spatiotemporal summation applied to the whole visual field, but research at Lund and elsewhere suggests that there is more going on in an insect's visual system. Insects make different trade-offs for different parts of the image, perhaps even optimising vision in each part independently. In theory, one region on an insect's retina could sum incoming photons for long periods to pick out the details of a flower head, say, while the rest could be optimised to probe the shadows for motion by increasing the number of photoreceptors from which signals are pooled (Journal of Vision, vol 9, p 1).
Warrant and his team call this mechanism "local adaptive spatiotemporal smoothing". It has become the inspiration for a new kind of digital image-processing algorithm, developed jointly by Warrant, mathematicians Henrik Malm and Magnus Oskarsson, also at Lund, and engineers at Toyota Motor Europe.
To understand how it works, imagine a single image recorded by a digital video camera. It is stored in the camera's memory as a rectangular array containing hundreds of thousands of numbers, each one representing the light intensity recorded by a pixel on the camera's light detector chip. The first step in the team's algorithm is a variation on a fairly standard image-processing technique that enhances brightness and contrast. A typical image recorded at night will contain lots of pixels with very low numbers near zero and maybe a few with very high numbers (around bright lights). The first step is to expand the range of the very low numbers, while compressing the range of the brightest numbers. This "non-linear amplification" brightens those parts of the image containing most information while leaving other parts almost unaltered. This is analogous to what happens as the neural signal passes from the photoreceptor cells in an insect's retina to the next layer of processing cells, says Warrant.
For an image taken in near-total darkness, this enhancement can reveal some detail, but it leaves the image grainy and speckled because any stray photons and electronic noise will also be amplified. The next step - spatiotemporal noise reduction - is designed to tackle this. According to Malm it is the part most directly inspired by insect vision.
It works by pooling the signals in neighbouring pixels both in space and time, while automatically looking for clues to determine the optimum amount of pooling for each part of the image. Mathematically, this is the equivalent of setting the best exposure time and pixel size independently for every spot in every frame.
To do this, the algorithm compares the value stored in each pixel with the values in neighbouring pixels and looks for any continuity or correlations across the frame, as these signify real objects (see diagram). Meanwhile any signals which occur in just one or two pixels in a single frame are most likely noise and can be removed.
The algorithm also looks for patterns repeating in successive frames to spot movement, and depending on the presence and speed of movement, it tunes the degree of temporal and spatial summation between pixels. Where an object is fast moving, summation between frames must stay low to avoid blurring, and the algorithm relies more on spatial summation within single frames. When objects are static the algorithm can pool in time and capture more spatial detail. Pooling pixels in this way comes at a price, though. "The fineness of spatial detail declines," Warrant says, "but the coarser details left can be seen much more clearly."
The final step in the algorithm helps sharpen the image by restoring edges that may have been lost or blurred in the noise-reduction processes. Sharpening is already widely used by many image-processing algorithms but the team found the process was more efficient when combined with the summation step. It compares brightness levels in neighbouring pixels, and where it spots a discontinuity between regions it uses a principle called lateral inhibition, also found in the visual system, to increase the light intensity on the bright side of the boundary, and decrease it on the darker side. This process helps define the form of objects, even if they or the camera are moving.
- New Scientist
- Not just a website!
- Subscribe to New Scientist and get:
- New Scientist magazine delivered every week
- Unlimited online access to articles from over 500 back issues
- Subscribe Now and Save
If you would like to reuse any content from New Scientist, either in print or online, please contact the syndication department first for permission. New Scientist does not own rights to photos, but there are a variety of licensing options available for use of articles and graphics we own the copyright to.