The former HSBC bank building on the corner of Bute Street and James Street is due to house 16 flats including a four-bedroom penthouse apartment.
Cardiff planners said the proposals would help regenerate the area.
The Victorian Society opposed the plan, saying it "would cause a high degree of unjustified harm" to the building.
The four-storey building was constructed in 1874 for coal owners and exporters Cory Bros and was altered and extended in 1914.
The building has been empty since 2014 and planners, in their report recommending approval to councillors, said the proposals were "welcomed in principle as the development would revitalise a large long-term vacant listed building".
They said the changes to the roof showed "appropriate regard to the special interest of the building".
But The Victorian Society had objected to the plans - particularly the penthouse - saying it would "cause a high degree of unjustified harm to the listed building and surrounding Mount Stuart Square Conservation Area".
It said the building "proudly and splendidly evokes the extraordinary wealth Cardiff enjoyed at the end of the 19th Century".
Cardiff council's planning committee passed the proposals on Wednesday subject to conditions and the development starting within 18 months.

The Dudley Muslim Association has offered Dudley Council £325,000 to drop court action in a dispute about the site where the mosque is to be built.
If accepted, the association would retain ownership of the land on Hall Street, and the council would not invoke its buy-back clause.
The council has so far incurred £213,000 in legal costs.
The land at the centre of the dispute was bought by the association 10 years ago, with the aim of replacing its current building on Castle Hill, which it said it had outgrown.
Proposals include a community and training centre, sports facilities and a two-storey car park, but the mosque itself and its three prayer halls have been the focus of opposition.
Branded at times a "mega mosque", the plans have attracted EDL supporters from across the country, although community leaders have in turn accused them of "scaremongering".
At its meeting on 25 June, the council's cabinet will refer the settlement offer to its scrutiny board for consideration.
Pete Lowe, leader of the council, said both the authority and the association were keen to avoid further costly legal disputes.
"We are carefully considering that offer in the interests of everyone concerned," he said.

Sweeney, 20, made 16 appearances for Rovers during a loan spell last season, while Telford, also 20, moved to Stoke from Blackpool in July 2015.
"I am now looking forward to playing in League One with Rovers and can't wait to get started," Telford said.
"I am really pleased to be coming back to Bristol Rovers," added Sweeney.
"I know that there were other clubs interested in taking me on loan, but once I knew that Rovers were interested there was only one club I wanted to join."
Sweeney was named as Young Player of the Year at Bristol Rovers last season.
Manager Darrell Clarke said: "I am looking forward to working with both of them as the season progresses and would like to thank Mark Cartwright and Mark Hughes at Stoke for allowing Ryan and Dominic to join us."
The Gas begin their season on Saturday with a trip to The Valley to face Charlton Athletic.
Find all the latest football transfers on our dedicated page.
Plans to turn a Grade II-listed building in Cardiff Bay into a restaurant and flats have been given the go-ahead by the council.
[NEXT_CONCEPT]
An out of court settlement has been offered by a Muslim group to end a long-running dispute over a new mosque.
[NEXT_CONCEPT]
Bristol Rovers have signed defender Ryan Sweeney and striker Dominic Telford on season-long loan deals from Premier League side Stoke City.