Because banks in the 1950s and 1960s would not lend money in African-American areas, the homes that went up were small. So family life took shape outdoors, among neighbors, said Ms. Granger, who has spent summers in Sag Harbor Hills since 1951.  “There was no television,” Ms. Granger said, describing life in the hamlet. “You got a little bit of radio. And so you really went in the homes to sleep and eat, and the rest of the time you were outside doing things.” Neighbors watched one another’s children, she said. Houses tended to stay within families or among friends; turnover was rare.  With the arrival of celebrities like Harry Belafonte, Lena Horne, the restaurateur B. Smith and Allan Houston of the New York Knicks, the communities rivaled Oak Bluffs on Martha’s Vineyard as a vacation mecca for successful African-Americans. Colson Whitehead, writing about the summers of the black elite in his novel “Sag Harbor,” described an oasis apart from his white prep school. “We fit in there,” he wrote.  At a village trustees’ meeting this month, residents faced off. Mr. Bronster was seeking approval to build a house of 5,300 square feet on several combined lots, the biggest in an area where the median house size is 1,378 square feet. Neighbors, including a group called Save Sag Harbor, which formed to keep big-box stores out of downtown, came ready to oppose him.  Mr. Bronster brought backup to the small-town gathering.  “He brought his architect, he brought a land-use attorney and some other attorney, and he had aerial photographs and professionally done graphs,” said Victoria Sharp, a former director of the AIDS center at St. Luke’s-Roosevelt Hospital in Manhattan.  “If Bronster was putting up a house of 4,000 square feet, people would be fine with that,” Ms. Sharp said. “But putting up a house of 53 hundred square feet represents a tipping point. The next one is 55, and the next one is 59. And he’s got big money behind him and is going to make this neighborhood what it never has been.”  Like others interviewed for this article, Ms. Sharp, who is white, said the tensions were not racial. She said that since moving to Sag Harbor Hills from another neighborhood in the village last February, she has been welcomed into her neighbors’ homes. “It’s exactly why I moved here,” she said. In 14 years in her previous house, she was never invited to a neighbor’s party. “There’s a real sense of community here. And that’s about, unfortunately, to be interrupted.”  Mr. Bronster said he had no intention of changing the community’s atmosphere. He, too, was drawn by the neighborly feel, he said.  “I’m especially appreciative of the history of the neighborhood, how it was started, what it means to the residents who live there and how that has enabled the community to develop a very warm and gregarious and welcoming sense,” he said. “That’s why I want to be there.” He added that he was developing dilapidated houses to rent to “year-round families that want to be part of the community.”  An even bigger house, of 5,900 square feet, has been proposed for four combined lots that belong to Robert Kapito, the president of the investment firm BlackRock, who is also one of Mr. Bronster’s investors. Mr. Kapito earned more than $20 million last year, according to company filings. Final approval for the two houses is pending.  Renee Simons, who lives next to the site of Mr. Kapito’s proposed house, said that she felt dismissed by the newcomers, and at odds with some neighbors, who hope the new houses will raise the value of their homes.  “It feels like us versus them, which is not healthy,” Ms. Simons said. “It’s not what I come here for. There’s such an increase in contention now.”  “We’re on defense,” she said. “So then you get called a rabble-rouser.”  Dianne McMillan Brannen, a real estate agent who has lived in Ninevah for 25 years, said she worried about a domino effect: investors combining lots to build bigger houses, which drives up sales prices, which tempts more families to sell, until eventually a historically rare African-American haven looks like just another upscale beach resort. In the last year, she said, 13 houses have been sold to builders or investors, compared with the usual four or five.  “An identity is the most important thing that could be lost,” Ms. Brannen said. “This area is not always going to be African-American. You could have 250 homes, and it would be an entirely different set of people here. And those sets of people are not going to regard it as we have regarded it. And they’re not going to have a story to tell about what’s happened previously. And unless we tell that story, it’ll go away.”  Mr. Pickens, whose grandfather was an early field secretary of the N.A.A.C.P., said he welcomed newcomers, whatever their race, but not investors or people with four or five houses and no commitment to the community.  “This is sort of reverse integration,” he said. “That’s fine, that’s the American way. But there are 5,000 miles of coastline in America, and five are commanded and owned by blacks. So we treasure what we own. That begins to disappear. Think about that. So that’s what we’re dealing with. And once you leave here, you can’t afford to come back.”  Of the newcomers, he asked: “Do they really want to be here or do they want to see us out of here? I’m for integration, I’m not for elimination.”  A version of this article appears in print on August 26, 2016, on page A1 of the New York edition with the headline: Black Enclave’s Newcomers: Mr. and Mrs. L.L.C. Order Reprints| Today's Paper|Subscribe
Residents of Sag Harbor’s African-American neighborhoods on Long Island worry that an influx of investors who are sweeping up properties will forever alter the community’s character.