The castle is Norman, with a large, square keep 113 feet high. It has been mistreated over the years: in 1215, King John - fighting the rebel barons who forced him to sign Magna Carta - used the fat from 40 pigs to set fire to the corner tower and undermine it! Even that wasn't enough to take the castle, and he had to starve the inhabitants out. (By contrast, the king himself would die a few months later from over-eating).*
Today, little of the interior survives, although you can see arches and the internal well, but there are still impressive views from the top. Look out for Satis House: it got its name because when Elizabeth I visited, she passed judgement on the hospitality in one word, "satis" (enough). Charles Dickens, a resident of the town, borrowed the name for Miss Havisham's home in Great Expectations. (Confusingly, he modelled the actual building on nearby Restoration House).
After climbing all those stairs, you can reward yourself with tea at the Cathedral's tea rooms. Rochester Cathedral also dates back to Norman times, and is conveniently located next to the castle.
*Well, there is doubt whether he really died of a eating too many peaches - it could have been poisoned plums, poisoned beer, or dysentery.