The 32-year-old made the last of his 63 Test appearances in the 16-12 victory over Italy in Turin last month.
Hamilton made his Scotland debut against Romania in November 2006.
"I have loved every single moment of wearing the thistle and feel extremely humble to have done it for nine years," the Saracens player said on Twitter.
"Today is both a sad day and a day that I feel extremely proud to have had.
"Thank you to the Scotland supporters for making it so special."
Hamilton was one of 11 players to miss out when Cotter cut his 42-man training squad to his final selection on Tuesday.
John Barclay, Blair Cowan, Adam Ashe, Hugh Blake, Chris Cusiter, Alex Dunbar, Rob Harley, Ruaridh Jackson, Duncan Taylor and Greig Tonks were also omitted.
Scotland open their World Cup Pool B campaign against Japan in Gloucester on 23 September.
They then face United States in Leeds before meeting South Africa and then Samoa in Newcastle.

He passed away peacefully in hospital on Tuesday after a short illness.
Born in Tourmakeady, County Mayo, he worked as a teacher before securing a part in the premiere of the Brian Friel play Translations in 1980.
Lally became a household name in Ireland for his role as Miley Byrne in the RTE soap opera Glenroe and later starred in the BBC series Ballykissangel.
He also appeared in the Hollywood movie Alexander and provided the voice for the Oscar-nominated, animated Irish film, The Secret of Kells.
As a fluent Irish speaker and advocate of the language, Lally had roles in several Irish language films.
He is survived by his wife Peggy and their children Saileog, Darach and Maghnus.

He said the Bank took "timely, comprehensive and concrete" action, which acted to "support, cushion and help the economy to adjust".
Last month's rate cut to an historic low of 0.25% helped support house prices and the wider economy, he added.
He added he was "absolutely serene" about the Bank's preparations for the impact from the Brexit vote.
Mr Carney was being questioned by MPs on the Treasury Select Committee about how the Bank handled the referendum outcome and its decision to cut interest rates to a record low.
The Bank was criticised by Brexit supporters for saying the economy would be hit from a vote to leave the EU.
But Mr Carney said the Bank's decisions had so far been "validated".
The "extraordinary preparations" made by the Bank "cushioned" the economy and will help the UK "make a success of Brexit", he told MPs on the Treasury Select Committee.
He said the Bank's actions were made in an effort "to help make the leaving of the EU a success as quickly as possible".
"This financial system, under the oversight of the Bank of England, sailed through what was a surprise to the vast majority of financial market participants," Mr Carney said.
MPs questioned whether the Bank was too aggressive in supporting the economy after the vote.
The UK's services, manufacturing and construction industries have all showed signs of recovery in August after taking a hit in July.
However, Mr Carney said he was "absolutely comfortable" with the Bank's decision to cut interest rates to 0.25% and pump billions of pounds into the economy.
He added that it is "welcome there is a rebound" in the economy.
Mr Carney said there has been quite a considerable improvement in mortgage borrowing costs "and we are seeing pass through of our actions".
When Labour MP Rachel Reeves raised concerns that the majority of lenders had yet to pass on the full cut in rates, Mr Carney said the Bank expected "virtually" all the rate cut to be passed on to borrowers in the next few months.
Scotland lock Jim Hamilton has announced his retirement from international rugby after he was left out of Vern Cotter's World Cup squad.
[NEXT_CONCEPT]
One of Ireland's best known actors, Mick Lally, has died at the age of 64.
[NEXT_CONCEPT]
Bank of England Governor Mark Carney has defended his action to mitigate the impact of Brexit.