The entire argument of John Stuart Mill’s On Liberty is summarized in the 12th paragraph of the “Introductory” chapter:
But there is a sphere of action in which society, as distinguished from the individual, has, if any, only an indirect interest; comprehending all that portion of a person’s life and conduct which affects only himself, or if it also affects others, only with their free, voluntary, and undeceived consent and participation. When I say only himself, I mean directly, and in the first instance: for whatever affects himself, may affect others through himself; and the objection which may be grounded on this contingency, will receive consideration in the sequel. This, then, is the appropriate region of human liberty. It comprises, first, the inward domain of consciousness; demanding liberty of conscience, in the most comprehensive sense; liberty of thought and feeling; absolute freedom of opinion and sentiment on all subjects, practical or speculative, scientific, moral, or theological. The liberty of expressing and publishing opinions may seem to fall under a different principle, since it belongs to that part of the conduct of an individual which concerns other people; but, being almost of as much importance as the liberty of thought itself, and resting in great part on the same reasons, is practically inseparable from it. Secondly, the principle requires liberty of tastes and pursuits; of framing the plan of our life to suit our own character; of doing as we like, subject to such consequences as may follow: without impediment from our fellow-creatures, so long as what we do does not harm them, even though they should think our conduct foolish, perverse, or wrong. Thirdly, from this liberty of each individual, follows the liberty, within the same limits, of combination among individuals; freedom to unite, for any purpose not involving harm to others: the persons combining being supposed to be of full age, and not forced or deceived.
The first key concept here is of a personal sphere that an individual cares about more than other people do–or where other people care mainly if they have freely chosen to interact with that individual. In the middle of the description of that concept is an important phrase: “free, voluntary and undeceived consent.” It is not enough to establish valid consent that it be free and voluntary. It must also be undeceived.
The delineation of this personal sphere is the hard part. John Stuart Mill identifies what goes on inside one’s own brain as the most obviously personal sphere. It then requires a deep set of arguments to argue that freedom of thought requires freedom of speech–that we can’t really be who we freely, voluntarily and undeceivedly choose to be even in our own minds unless there is freedom of speech. For one thing, it is easier to think something if one dares to express it to others. For another without other people having freedom of speech, one will have relatively few choices of things to think.
Moving beyond speech to action, the delineation of a personal sphere ultimately requires reference to the physicality of human beings and ownership of objects. There are various ways to delineate this personal sphere. But as long as the principles are followed of giving each person a roughly equal personal sphere, without too much overlap, it will probably be all right. There may even be some ownable things that are “frills” and therefore can be distributed unequally as incentives without compromising people’s lives.
Finally, the idea that people can choose to pool their personal spheres for certain purposes to get together as a group and do their thing is very important. This is one of the most contested areas of the concept of liberty. For example, in current US law, getting together as a group for consensual sexual activity is a liberty that is now quite well protected, but getting together as a group for consensual economic activity is much less well protected–in part because many consensual economic activities that provide surplus to both parts still have a strong adversarial dimension because of disagreements over the relevant wage or price that divides the surplus. Debates about the scope of group liberty rage, with some arguing that sexual freedom should be curtailed, and others arguing that economic freedom should be expanded, while others argue for the status quo of freedom for consensual sexual activities and restraints for consensual economic activities.
Here, it is worth noticing that every argument for how two parties conducting consensual economic activity might affect others has a counterpart argument for how two parties conducting consensual sexual activity might affect others and vice versa.
The moral is that liberty is wonderful, but not simple. It takes deep thought to understand liberty.