In 1895, the astronomer Simon Newcomb published his "Tables of the Sun," based on observations of the sun's position from 1750 to 1892. (http://en.wikipedia.org/wiki/Newcomb%27s_Tables_of_the_Sun) These calculations turned out to be reliable enough that astronomers continued to use them until the 1980s.
Before 1956, the second was defined as the mean solar second, or in other words, 1/86,400 of the time the earth takes to spin around on its own axis and see the sun again each day. But because the moon's gravity and the tides are slowing down the earth's spin, this is not a stable quantity.
From 1950-1956, the international authorities agreed to redefine the second to be the "ephemeris second," based on the speed of the earth's orbit around the sun in 1900, as predicted in Newcomb's tables. The earth's orbit around the sun is not slowing down, at least not on anything like the effect on the earth's spin around its own axis. (In practice the ephemeris second is measured by looking at the moon's orbit around the earth and taking pictures of what stars the moon is near.)
Because Newcomb's tables cover observations from 1750 to 1892, the "ephemeris second" corresponds to the mean solar second at the middle of this period, or about 1820. (http://tycho.usno.navy.mil/leapsec.html)
Meanwhile, from 1952 to 1958, astronomers from the U.S. Navy and the British National Physical Laboratory measured the frequency of cesium oscillations in terms of the ephemeris second. (http://www.leapsecond.com/history/1958-PhysRev-v1-n3-Markowitz-Hall-Essen-Parry.pdf) Cesium is even more stable than the orbit of the earth around the sun.
There are a few ways to do the calculations that they show in the paper (having to do with exactly what period they observed over and whether they corrected for some subtleties re: the moon's orbit), giving results between 9,192,631,761 and 9,192,631,780. The average was 9,192,631,770.
In 1967, this became the official definition of the SI second, replacing the ephemeris second. But the reason the number is what it is is because Newcomb analyzed observations from 1750 to 1892, and the middle of that period is 1820, and that's how fast the earth was spinning on its axis in 1820.