This is a guest post by Martina Mustroph, one of Greta’s top student writers for Spring 2007.
When you’re typing, your senses of touch, hearing, and sight align. You feel, see, and hear your fingers touch the keyboard. Now imagine that you are outdoors and you feel a drop of water hit your hand. If you are like me, then it probably immediately occurs to you that it was a raindrop, so you stretch out your hand to see if more will come, and you look up at the sky for menacing clouds. Let’s say the sky is blue and clear as far as you can see. Now your senses of touch and sight are at odds: your sense of touch just told you it was raining, but your sense of sight said it was not. In this case, you don’t go running for cover; you choose to go with the information you get from your sense of vision and not the information you got from your sense of touch, probably because you only felt that one drop.
But what if you don’t get much more information from one sense than from the other? A team led by Jean-Pierre Bresciani showed people flashes on a screen. At the same time, a device tapped these people’s right hand a certain number of times. The screen was set up so that it obscured people’s right hand from their field of vision, and the flashes occurred where the right hand would be. People could only feel–but not see–their hand being touched. People were instructed to either count the number of flashes they saw on the screen or the number of taps they felt on their hand, but they were never told to pay attention to both simultaneously. The number of taps given differed from the number of flashes by plus or minus one. Why just one? It goes back to the rain example: when you have tons of information (miles of blue clear sky) from one sense telling you something and little information (a raindrop) from the other sense, it’s easy to pick one sense over the other. Likewise, if you are shown 2 flashes but feel 5 touches, it’s easy to dismiss the information you get from one sense over the other.
Now let’s look at how people performed. First, people were only given taps or only given flashes to see how much their counts would vary when only one sense was stimulated. People’s counting of taps was less variable than their counting of flashes. Here we need to distinguish between variability and accuracy. If Tom is repeatedly shown 2 flashes but keeps counting 3 every time, Tom’s count is reliable (it has low variability), but that doesn’t make it accurate. Variability and accuracy are not the same thing. Keep in mind that at this point, people are only shown flashes or are receiving taps. When this is done, touch is the more reliable sense. Then people were given both flashes and taps simultaneously. They were never told to pay attention to both flashes and taps, yet apparently, people do automatically pay attention to both. How do we know this? Even though they were told to focus just on the taps or just on the flashes, their accuracy of counting changed whenever the second number of events differed from the one they were focusing on (in other words, the added taps or added flashes messed them up). Take a look at this graph of the results:
The graph shows just one case: when three flashes were shown. In this case, as you can see, people believed they saw more than three flashes when they were tapped four times, and less than three flashes when they were tapped twice, so the number of taps affects the perception of flashes. The pattern also worked when two or four flashes were shown. When people are given one more tap than the number of flashes, suddenly their count of the number of flashes rises, once again even though the actual number of flashes they saw did not change at all. If the taps were not messing people up, the line on the graph should have a slope of 0; it would be a straight horizontal line.
Did their counting variability also change? Yes! Take a look at this graph:
People are significantly more consistent (although less accurate!) at counting the number of taps (but not flashes) when they are given flashes to go along with the taps-and here is the amazing part–even when the added flashes differed in number from the taps they were counting! This means that we are automatically processing the added background stimulus, because if we weren’t, people’s counts should not change when the second stimulus is added. Yet the range in the reported numbers of perceived taps decreases when both flashes and taps are given simultaneously.
One final thing: At the start of the experiment, Bresciani and his team found that when people are only shown flashes or only given taps, their counting is very reliable. When people are shown both flashes and given taps at the same time, there is another interesting thing that happens. Even though you’re paying attention to flashes, feeling a tap messes you up. It only sort of works the other way. The effect of touch on vision is more pronounced than the effect of vision on touch. When you’re paying attention to taps, vision only rarely messes you up. Seems strange, right? Actually, that’s explained by touch being the more reliable sense. We appear to give more weight to the more reliable sense. The fact that vision, the less reliable sense, still affects people’s counting of touch, the more reliable sense, means that we automatically process both, but then treat each sense with a weight corresponding to its relative reliability. If we just blocked out the information we get from one sense, then when counting taps, people should not be less accurate when flashes are added, but their count is affected by them, meaning that they do process the flashes.
It seems like this is the optimal way for our brains to work. Going back to our rain example: Ever felt a drop of water land on your hand and look up to indeed see a few clouds overhead that you just hadn’t noticed? In this case, you conclude that it may indeed rain soon. What you really did in coming to that split-second conclusion is combine information from both senses to assess the situation in front (or above) of you, and that combination of information from both senses helps you come to an appropriate decision.
Bresciani, J-P., Dammeier, F., & Ernst, M.O. (2006). Vision and touch are automatically integrated for the perception of sequences of events. Journal of Vision, 6, 554-564.