Biography has come a long way. In particular, it can claim to be as ancient as any other written form - poetry, for instance, with which it is inextricably linked. It has also been powerfully influential. Most of the world's great religions have a biographical element: at the core of Christian teaching are four resonant biographies.
Biography has had its ups and downs over the last few millennia, but today - ostensibly - we live in a biographical golden age. No literary genre is more sought after. Many upper-brow members of the public, when asked what they read, say "biographies" without being very specific about subject area. The well-researched life seems to have an appeal to people who are made impatient by mere fiction: perhaps it is Protestant self-improvement and voyeurism wrapped into one. That biography is popular, however, is beyond dispute.
Yet biography's position remains uneasy. There is a kind of person who sneers at biography. There are several reasons for this. The most obvious is that many biographies do not live up to the claims made for them. Because publishers, newspapers, agents are enthusiastic about biography, many writers - novelists, say, or poets with little biographical talent - try their hand, with embarrassing results. Hence biography has a mixed reputation among writers and critics who have not, themselves, joined the gold rush and who claim that many biographies are badly written, factually sloppy and hagiographical.
At the same time, there is a philosophical objection. This is the harder objection to deal with. "Creative" writers and analytic historians maintain that biography is neither chalk nor cheese: limited in imaginative range, and of its nature insufficiently grounded in the historical method. More than 40 years have elapsed since EH Carr issued his classic and superficially persuasive denunciation of biography in What is History? Meanwhile there have been others in the Cambridge radical tradition who have been similarly scornful. Eric Hobsbawm dismisses biographies as "Victorian tomes". Provocative echoes of this are to be found in the essays of David Cannadine.
At the same time, the literary establishment, with powerful English faculty professors as its gatekeepers, has woken up to the role of biography, but draws an unspoken distinction - treating "biography" and "literary biography" (ie biographies of writers) as synonymous terms, and often ignoring non-literary biography altogether. Thus there is a growing academic literature on "biography" that takes no account of the biographies of scientists or statesmen.
In a way, this prejudice - conscious and often unconscious - is odd, and not just because many of those who deride biographies nevertheless read them. It cannot be dismissed as merely snobbish; neither is it just the product of a progressive movement in favour of emphasising social movements over "chaps" - the old-fashioned stress on the role of the individuals. The simple sociology-mindedness of one school of history has long been displaced, in our so-called postmodern world, by a greater tolerance of variety.
Probably, it was never philosophical at all, but merely reflected the sad reality that most of the talked-about biographies lacked the qualities of imaginative insight and rigour available in other disciplines; and the number of biographies worth reading for their own sake, rather than for the information conveyed, was and remains small.
Are there as many low quality novels and history books? Possibly. It could be, however, that one reason for resentment is that biography has an unfair advantage. Where a novelist has to create a world and grab attention within the first few pages, a biographer can compensate for dull thoughts and flat writing by offering a stolen yarn about a figure of historical interest.
A second reason is more generic. Sometimes, the popularity of a product leads to change. At other times, it has the opposite effect. Publishers and agents join together: if this is what the public likes, let them have more of it - and that is what is supplied. Hence one cause of irritation about biography in the hands of many earnest practitioners is that it has become a constipated form. Ever longer, ever better researched biographies are becoming scholarly monuments - to be admired and surreptitiously skimmed, rather than read in toto as cohesive works of art. The modern biographer with ambition (and, probably, an academic career) likes to be "definitive", which tends to mean inclusiveness rather than selectivity: a definitive biography is seldom a short one.
So much for the coolness which encourages serious intellectuals to place biography on the fringes of historical or critical writing. Since the earliest Hittite and Chinese inscriptions, and more particularly since Plutarch, writing about the lives of heroes and villains has passed restlessly through many phases. It is fair to believe that it will last at least as long as the parvenu novel. But where is it heading?
Of course the accusation of immobility is not to denigrate biography as such, or even many of the books in the dock. On the contrary: few would dispute that the boom of the last 20 years has done an enormous amount for the quality of biographies, or that the best half-dozen published each year compare favourably with all but a handful of star volumes published a generation ago. Both the scholarship and scholarly apparatus of the contemporary study of a life have improved beyond measure, and some of the old gripes - about purple passages in which the author attempts to read the subject's mind - have largely been disposed of by new biographers who have no truck with anything but source-based data.
This is all to the good. What is not so good is the formulaic nature of even the best of contemporary biographical writing, the tight defensiveness of the genre in the face of the kind of criticisms mentioned, and the lack of experimentation. There are a few wild cards, playing around with chronology. Some writers have tried group biography. None, however, has yet succeeded in becoming an effective revolutionary, or has made a particularly convincing attempt to do so.
Indeed, there is a strong argument for saying that, despite Lytton Strachey and a century of supposed biographical iconoclasm, the Victorian tome still reigns. Most biographies are as explicit as possible about private peccadilloes. But almost all remain judgmental, seeing as a key objective the enchantment of the subject's reputation. This is particularly true of literary biography. Indeed, the conscious or unconscious subtext of virtually all literary biography is that such-and-such hitherto obscure playwright can be excused philandering or domestic violence or other departures from bourgeois moral norms, because of the value of the work.
And so it goes on: biographies get fatter, more professional, harder for the struggling would-be life writer to compete with, given the time and stamina required. Indexes, prefaces, references and explanatory notes expand. Care over artwork and cover design increases, along with prices, and desirability as Christmas presents. Will the trend continue indefinitely, as the public pocket gets deeper and the seasonal appetite for big, solid, meticulous non-fiction continues to grow? Common sense suggests that a change must come. Given the conservatism of the readers and writers of biographies, the change may be gradual. The big biographies will continue, but some of them may become sleeker. Meanwhile, the public focus may already be shifting a little - away from solidarity and towards biographical imagination.
To say this is not to make a nostalgic plea on behalf of the "evocative" or "atmospheric" biography, which places intuition above learning. It is more to note an emerging taste for the book about a famous person that does not require 300 pages of references to make its points, but - on the contrary - makes every telling phrase and word count. Roy Jenkins, author of long, brilliant books about Gladstone and Churchill but also of tight essays of selected leaders, comes to mind. So does Francis Wheen's wittily demystifying Karl Marx. Such books show that the scope, in skilled hands, of biographical writing and the insights it can offer into the human condition are really limitless, and that biography need not regard itself as anybody's poor (or nouveau riche) relation.
Neither should biography be seen as a hybrid, occupying an intellectually untenable position between fact-based history and true creative writing. Biography is itself. What a biography ought to be like is of course an unanswerable question, although biography in the modern sense operates within fairly tight rules - attention to accuracy, avoidance of suppressio veri most important among them, and a recognition that there is no such thing as a "true" biography: however scrupulous the research, nobody has access to another's soul, and the character on the page is the author's unique creation. One aspect of the creativity is the subject-in-context and it is this that makes the complaint about over-emphasising the role of the individual so off-beam.
Indeed, far from underplaying social factors, the good biographer highlights them, to give added precision to the story. Good biography is flexible, making unexpected connections across periods of time, and including unexpected essays on topics which, for the involvement of the subject, might never get written about at all.
The metaphor of portraiture is appropriate here. A good biography is like a good portrait: it captures the essence of the sitter by being much more than a likeness. A good portrait is about history, philosophy, milieu. It asks questions as well as answering them, brushstrokes are economical and always to the subtlest effect. Think of Velázquez, Sargent, Freud. Biography can be like that.
· Ben Pimlott died in April. This essay appears in Lives for Sale: Biographers' Tales, edited by Mark Bostridge and published by Continuum on September 1 at £16.99