The order castle was finished before the Livonian War (1558–1581). The wooden constructions of the castle were burnt in 1558, when the castle was sieged by the Russians. They reconstructed the castle in 1570 and used it until 1581, when the Swedish troops by the lead of Pontus de la Gardie captured it. By his order the castle was partly demolished but was renewed in 1605–1608 and served as a Swedish stronghold until 1617, when it was given in the possession of field marshal Hermann Wrangell by king Gustav II Adolf. Finally, the castle was blown up by Swedes during another Russian-Swedish war in 1658, so it would not fall into the hands of the Russian troops that looted the area.
Today the castle belongs to the Museums of Virumaa.
Toolse Castle (Tolsburg) was probably built to defend the harbor on the Toolse bay, that existed in the 13th–14th century. In those days the area with the harbor, and house-castle in the northern part of Toolse peninsula belonged to noble family Wrangell.
The exact time of constructing the house-castle is unknown, but it existed already in 1471, when on the order of the master of Livonian Order Wolthus von Herse, the peninsula with the house-castle was expropriated from the Wrangells and the order castle Vredeborg (later Tolsburg) was built instead.