Harlem was “a synonym for elegant living through a good part of the nineteenth century.”The village remained largely farmland estates, such as [Conrad] Van Keulen’s Hook. Otterspoor, bordered north of the Mill Creek (now 108th St., orig. Montagne Creek at 109th St.), which flowed into Harlem Lake, to the farm of Morris Randall, northwest on the Harlem River, and westward to the Peter Benson, or Mill Farm. This former bowery of land was subdivided into twenty-two equal plots, of about 6 to 8 acres each, of which portions later owned by Abraham Storm, including thirty-one acres (east of Fifth Avenue between 110th & 125th St.) were sold by Storm’s widow Catherine in 1795 to James Roosevelt (great grandfather of President Franklin D. Roosevelt, 1760–1847). This branch of the Roosevelt family subsequently moved to the town of Hyde Park, but several of Roosevelt’s children remain interred in Harlem.
As late as 1820, the community had dwindled to 91 families, one church, a school, and a library. Wealthy farmers, known as “patroons”, maintained these country estates largely on the heights overlooking the Hudson River. Service connecting the outlays of Harlem with the rest of the City of New York (on the southern tip of the island of Manhattan) was done via steamboat on the East River, an hour-and-a-half passage, sometimes interrupted when the river froze in winter, or else by stagecoach along the Boston Post Road, which descended from McGown’s Pass (now in Central Park) and skirted the salt marshes around 110th Street, to pass through Harlem.
The New York and Harlem Railroad (now Metro North) was incorporated in 1831 to better link the city with Harlem and Westchester Co., starting at a depot at East 23rd Street, and extending 127 miles north along Lenox Avenue to a railroad junction in Columbia County at Chatham, New York by 1851. Charles Henry Hall, a wealthy lawyer and land speculator, recognized the changes that this railroad would make possible in Harlem and began a successful program of infrastructure development, building out streets, gas lines, sewer lines, and other facilities needed for urban life. Piers to the west of Harlem were also built, enabling Harlem to become a center of industry serving New York City. The rapid development of infrastructure enabled some to become wealthy, and the area became important to politicians, many of whom lived in Harlem. New York mayors Cornelius Van Wyck Lawrence and Daniel Tiemann both lived in Harlem in this period. For many in New York City, Harlem was at this time regarded as a sort of country retreat. The village had a population of poorer residents as well, including blacks, who came north to work in factories or to take advantage of relatively low rents.
Some of the land became occupied by Irish squatters, whose presence further depressed property values (photo above Squatter Settlement in Harlem at 7th Avenues and 137th Street, around 1858).