In the era before banking was invented, where did wealthy Romans store their cash? The emperor probably could store it somewhere in the imperial palace thanks to the Praetorian guard, but what about the others? Did they hire guards to protect their "silver room" or did they hide it?
Banking existed in the era of the Romans and earlier. In ancient Greece and Asia Minor temples served as a sanctuary where individuals could make deposits for safekeeping. This practice continued with the Romans (see this article titled "Temple Banking In Rome"). For instance, the Temple of Artemis in Ephesus was the largest depository in Asia and served as a depository for Aristotle, Caesar, Dio Chrysostom, Plautus, Plutarch, Strabo, and Xenophon. The Temple of Saturn in Rome housed the aerarium (public treasury). During the imperial period the Temple of Castor and Pollux in Rome was a depository for the State treasury.
The era of the Romans also saw the rise of private depositories. Private individuals (or individuals of a private organization) that served as bankers were known as argentarii. Such individuals were members of a collegium, or guild. Many individuals entrusted all their capital to them. The way these bankers transacted business is similar to modern times. Individuals could deposit funds with a banker, and pay their debts using a draft or check from the bank. If both the debtor and creditor had an account with the same bank it was easy for the bank to internally transfer the funds between the accounts.
There were two types of deposits that could be made with an argentarius (singular form of argentarii). A depositum did not pay interest and was simply a way for the owner to give someone else the responsibility of keeping it safe. The argentarius would not use a depositum in attempt to make money off it. A creditum paid interest to the owner and allowed the argentarius to attempt to profit from the creditum.
There were also public bankers, known as mensarii, who were appointed by the Roman government to loan out money in times of distress, with the first time being 352 B.C. I mention them because a class of the mensarii, the mensularii or nummularii (probably an inferior rank since mensularii is formed from a diminutive), seem to have been permanent state employees that private individuals may have deposited money with.
This page provides a good overview.
This page goes into detail and cites many primary sources.
This is an addition to Mike Rodney's answer. The Twelve Tables, traditionally written in 450 BC, were some of Rome's most ancient laws.
The majority of Table III deals with banking. In particular Law I says that bankers can't steal deposits; Law II forbids usury†; Laws V through X concern treatment of delinquent debtors.
So banking was common enough 2500 years ago in Rome that when the plebs demanded written laws they were some of the first few things that were regulated.
† Not more than uncia, variously interpreted as an annual rate of 10% or 12.5%.