Holly was born October 3, 1829 in Washington, D.C. He joined the Protestant Episcopal Church and became a deacon in 1855 and a priest in 1856.
Holly believed that African Americans had no future in the United States and the only answer was emigration. He was a delegate to the first NationalEmigration Convention of Colored People in 1854 and the next year represented the National Emigration Board as commissioner. Holly promoted emigration to Haiti and delivered a series of lectures that were published in 1857 as “Vindication of the Capacity of the Negro Race for Self Governance and Civilized Progress.”
In 1861, Holly led 110 African Americans to Haiti and the next year he became a Haitian citizen. In 1874, he was consecrated missionary bishop of Haiti and in 1878 he was recognized as bishop of the Orthodox Apostolic Church of Haiti.