You observed correctly.
The Second Temple was built upon Temple Mount. Around 20 B.C., King Herod the Great lavishly renovated the site, and in the process endowed the hill with a retaining wall. This structure became the only remnants of the Second Temple after its destruction in A.D. 70, most famously in the form of the Wailing Wall. In fact, however, Herod's walls encircled the whole of Temple Mount, and these formed parts of Jersualem's city walls.
In 1967 Israel capturd Old Jerusalem (and the Temple Mount) fom Jordan. It was found that the wall extended all the way around the Temple Mount and is part of the city wall near the Lion's Gate.
- Henty, George A. For the Temple: A Tale of the Fall of Jerusalem. Fireship Press, 2010.
Ancient Jerusalem possessed three sets of walls. Although their precise paths are disputed, based on the descriptions of Josephus in his The Jewish War we know they extended westward out of Temple Mount. The eastern walls of the Herodian Temple were thus also the city's eastern walls, while the remaining sides formed part of the inner city's fortifications.
The first wall embraced the two hills. It ... extended along the biblical City of David, encompassing the Temple and from there to the Tower of Antonia at the north-eastern corner of the Temple Mount, and along the Western Wall of the Temple Mount until the upper city. The short second wall encompassed the market places in the commercial quarter of the city ... down to Damascus Gate and back again to the Tower of Antonia.
- Abramski, Shmuel. Jerusalem in the Time of the Second Temple. Jerusalem: Department for Education and Culture in the Diaspora of the World Zionist Organization, 1968.
The Fortress of Antonia in particular stood on the northeastern corner of the Temple Mount and indeed played a part during the Roman siege of Jerusalem in A.D. 70. After entering the city by breaching the Third Wall, Titus then stormed Antonia, and from thence proceeded to capture the Temple and the rest of the city.