Brewer's: Upper Ten Thousand
or The Upper Ten. The aristocracy. The term was first used by N.P. Willis, in speaking of the fashionables of New York, who at that time were not more than ten thousand in number.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894