It comes in a shake-up of UK military buildings and resettling of regiments.
Brecon and Radnorshire Conservative MP Chris Davies condemned the closure, saying there had been a barracks in Brecon since 1805, home to troops who fought the Zulus at Rorke's Drift.
"This decision is abhorrent and I shall be fighting it every step of the way," he said.
"The government has a great deal of questions to answer over why it is proposing to close a well-loved and historic barracks in a vitally important military town.
"Brecon Barracks has served our country with distinction over its long history, with soldiers from the site fighting in every conflict since the early 19th century.
"This decision shows a blatant lack of respect for that history."
Mr Davies said he was launching a petition against the decision, saying the Brecon area had some of the highest unemployment levels in Wales.
He also hoped the closure would not damage the town's "thriving" military tourism industry.
Brecon barracks has about 85 civilian staff and 90 military but it is not thought jobs are at risk.
Mr Davies said he understood the nearby Sennybridge training ground and infantry school at Dering Lines would not be affected.
Defence Secretary Sir Michael Fallon told the Commons on Monday the reorganisation in Wales would see a specialist light infantry centre created at St Athan, Vale of Glamorgan.
Cawdor Barracks, Pembrokeshire - whose closure was previously announced in 2013 - will now shut in 2024, while a storage depot at Sennybridge will go in 2025.
Responding for Labour, Shadow Defence Secretary Nia Griffith, MP for Llanelli, said the ministry was "right to restructure its estate".
But she warned closing bases would affect the livelihoods of many people who would face "gnawing uncertainty" over their future.

The vigilantes were celebrating a successful operation with the military against the Islamist militants.
An improvised explosive device (IED) went off killing and injuring people gathered around the celebrations.
Despite losing territory this year, Boko Haram still controls a few areas.
The BBC's Nigeria correspondent Will Ross says although it seems to have been an accident, this tragedy highlights the danger that remains even after the jihadists have been flushed out of an area.
African news updates
"The vigilante group together with the military decided to go to villages in Marte local government [area] where there is preponderance of Boko Haram elements," MP Tahir Monguno told the BBC Hausa service.
"In the course of the operation they killed some Boko Haram insurgents and they confiscated their weapons," he said.
But one of the vigilantes failed to disclose one of the IEDs picked up during the two-day mission and took it back to Monguno, about 135km (83 miles) north-east of the Borno state capital, Maiduguri.
"Because of the successes they recorded there was celebration galore," the MP said, with many people joining in.
Ten people were killed instantaneously when the device exploded on Tuesday evening and another 13 people died by the time the injured were taken to Maiduguri, he said.
"Fifteen of them are [still] in the intensive care unit of the hospital," Mr Monguno said.
Will new military HQ defeat Boko Haram?
Why Boko Haram remains a threat
A military source told the BBC that 12 people transported to Maiduguri had died but was unable to confirm the number of those who died in Monguno.
Earlier, a witness told the Associated Press news agency that the device was found at an abandoned Boko Haram camp.
According to Amnesty International, at least 17,000 people, mostly civilians, have been killed since the group launched their violent uprising to impose Islamic rule in 2009.
The group is still holding many women, girls and children captive, including 219 schoolgirls it kidnapped from a school in Chibok in April last year.
With the help of Niger and Chad, Nigeria's military has been able to recapture most towns and villages taken by the group.
Despite losing territory, the militants are still active in the north-east and Chad blames Boko Haram for two suicide attacks in its capital, N'Djamena on Monday.
The Ministry of Defence is to close the British Army's Welsh headquarters at Brecon, Powys, in 2027.
[NEXT_CONCEPT]
At least 23 people have been killed in the north-eastern Nigerian town of Monguno after a Boko Haram bomb confiscated by vigilantes exploded, an MP and army source have told the BBC.