While working on a DC motor controller project I needed to decode quadrature encoder signals.
Basically a quadrature encoder has two signal lines, A and B. Rotational direction of the encoder can be determined by the phase difference between these signals.
A simple way to detect the phase difference of the signals with the ATMEGA168 is to generate an interrupt on the falling edge of signal A and look at the value of signal B. If signal B is high of the falling edge of signal A the encoder is turning one way. And if signal B is low on the falling edge of signal A the encoder is turning the other way.