How to determine cable sizes in theory
There are well established methods for calculating the required size of cables, based on:
I²R Heating - acceptable conductor temperature before the insulation material is damaged.
For regular PVC-insulated cables, you may assume this as 75°C (V-75.)
Other temperature ratings are possible. In Australia, mains power cable is often rated to 90°C - either because it is 90°C rated PVC, or because it's 90°V rated XLPE.
For special applications, you can get 110°C or 150°C rated cable. For really special applications, you can use metal insulated metal sheathed cable 'Pyrotenax', which is literally fireproof and can be run as hot as you please.
Heat dissipation of cable - depends on the thickness of the cable's insulation, outer surface area of the cable - basically, all the things affecting the cable's "thermal resistance".
Ambient conditions. Cables in circulating air are able to dissipate heat better than those where the heat is trapped inside an enclosure.
There are reports and standards that detail the theoretical calculations of how effectively cables dissipate heat under a variety of conditions. Try ERA report 69.30 Parts III and V for details.
How to select cables in practice
In practice, cable rating information is tabulated into a usable form like the examples below, from the Nexans Olex "Handbook 2013", a catalogue of low-voltage cables. This is simply a look-up table of current-carrying capacity (amps) vs. cable size.
Note all conductor sizes are specified in mm² - conversion to AWG or kcmil is left as an exercise to the reader.
Also note this is for low voltage cables in the sense of mains power cables (240 VAC or 415V three phase) but the same principles apply to extra-low voltage wiring. For heating purposes, the amps are all that matter. Whether the amps in question are 240 VAC amps or 5 VDC amps is largely irrelevant.
For de-rating purposes, you may assume that the inside of a sealed electronics enclosure is something like "enclosed conduit in air" - this is a plastic enclosure with no air circulation.
It is also normal to apply derating factors for the external ambient air temperature. These tabulated ratings are all at 40°C ambient temperature, which is the norm for Australia. A 45°C ambient temperature might translate to a 0.91 derating factor, for example.
Yet more derating factors apply if you are running multiple cables bunched together, as the cables heat each other up.
If you want to be conservative, take the 'enclosed, in air' cable rating, and apply a further 0.65 derating factor, which should cover you for pretty much any possible derating you will experience in practice, due to ambient temperature, cooling (or lack thereof), bunching of cables, etc.
For context, 1.5mm² is the normal size for domestic 10 A extension cables. 2.5mm² is the normal size for building wire (i.e. wiring supplying wall power points.)
You might consult your local electrical code for further guidance on cable derating factors. In Australia this is AS3008.1.1 Electrical installations - selection of cables. This has tables giving derating factors for various temperatures, various bunchings of cables, and so on.
In the US it might be one of your NEC codes.
Again, these standards are intended to be applied for mains power cabling, but the same physical effects apply to all cables - heating due to 240 VAC amps is the same as 5 VDC amps. There is some minor difference due to thicker insulation on higher voltage cables, which thermally insulates the cable - but this is conservative in your favour if you are considering low voltage DC cables.
You might notice the above gives no consideration to cyclic loading. If you need a wire to carry a cyclic load, try this table: