We investigate the formation of brown dwarfs and very low-mass stars through the gravitational fragmentation of infalling gas into stellar clusters. The gravitational potential of a forming stellar cluster provides the focus that attracts gas from the surrounding molecular cloud. Structures present in the gas grow, forming filaments flowing into the cluster centre. These filaments attain high gas densities due to the combination of the cluster potential and local self-gravity. The resultant Jeans masses are low, allowing the formation of very low-mass fragments. The tidal shear and high velocity dispersion present in the cluster preclude any subsequent accretion thus resulting in the formation of brown dwarfs or very low-mass stars. Ejections are not required as the brown dwarfs enter the cluster with high relative velocities, suggesting that their disc and binary properties should be similar to that of low-mass stars. This mechanism requires the presence of a strong gravitational potential due to the stellar cluster implying that brown dwarf formation should be more frequent in stellar clusters than in distributed populations of young stars. Brown dwarfs formed in isolation would require another formation mechanism such as due to turbulent fragmentation.