A few years ago, in a survey conducted by an accident prevention charity, 80% of respondents admitted to going through life on autopilot: arriving at the end of a car journey with no memory of driving there, buying the same item twice without realising, even turning up at the office on a day off. The other 20% must have been lying or deluded. (Or just answering the survey on autopilot, perhaps?) To an unnerving extent – made clearer by ongoing research – we're all creatures of habit, spending our days acting out ingrained behaviours and responses over which we exert no control. This has many advantages: if our brains weren't built to convert as many actions as possible into automatic routines, we would seize up trying to breathe or walk, let alone drive a car. But it's also frightening. Treading the well-worn paths of habit, we easily get mired in jobs, relationships or ways of thinking that make us miserable, in lives we'd never have consciously chosen. "Here you are, here we all are, semi-automated creatures in our tram-track worlds, running through the paths of least resistance," as Vincent Deary puts it, in one of two new books on how we get stuck – and on finding the will to forge new paths when life demands it.  Paul Dolan, an LSE economist and government well-being adviser – a man who knows his way around the Nudge Unit – has written the more businesslike of the two. The "science of happiness", he starts by pointing out, is full of bizarre and contradictory findings. Parents report that parenting makes life much more meaningful, yet seem to experience no more pleasure than non-parents; more money doesn't lead to more happiness, unless you frame the question differently, in which case it does. One problem, he argues, is that psychologists simply try to find out which "inputs" – income, work, marital status, age, religiosity and so on – are correlated with the "output" of happiness. But in fact happiness also depends on how we allocate attention to those things. Imagine two biscuit factories, one run well, the other incompetently: they might have identical inputs (sugar, flour, labour, electricity) yet produce very different quantities of biscuits, depending on their production processes. The same goes for manufacturing happiness. Attention, Dolan writes, "acts as a production process that converts stimuli into happiness". Attention is a scarce resource: give it to one thing, and by definition you can't give it to something else. If you're not as happy as you could be, "you must be misallocating your attention".  Dolan risks sounding as if he has reduced the age-old conundrum of human fulfilment to a maths exercise. But treating happiness as an attention problem goes a long way to explain why we aren't happier, and how we could be. Specifically, it explains how the "tram-tracks" of habit and ingrained thinking, along with various cognitive biases, sabotage us, by directing our attention to the wrong things. We eat unhealthy food because it's there. (The proximity of a school to a fast-food restaurant is correlated with obesity among pupils.) Or we lose touch with valued friends because it's easier to watch the TV. We focus on the happiness we'd get from a bigger house – only to find, once we've moved, that the misery of the increased commute outweighs the benefits. Students choosing between universities subconsciously notice whether it is rainy or sunny on open days, and let it influence their decisions. More generally, we pay attention to what we think ought to make us happy – to our lofty judgments about a "meaningful life" – instead of moment-to-moment feedback about which activities actually bring us feelings of pleasure or purpose.  And so we must redesign our mental factories, so that they produce more happiness. Partly, this is a matter of what you might call self-nudging, organising your life "in ways so that you can go with the grain of your human nature and be happier without having to think too hard about it". If you only stock your fridge with kale and tofu in the first place, healthy eating won't take much willpower. If you make a default plan to call your faraway friend at 3pm every Sunday, you'll co-opt inertia to your cause: the lazy option will be to stick to the plan. Dolan is especially illuminating when it comes to bigger life-choices, and how to get reliable feedback on what will truly make you happy. Don't ask yourself if you should take that new job; ask your friends, who'll be less blinded by indecision and anxiety, and phrase the question like this: "How do you think my day-to-day life will be in a couple of months if I take the new job?" In one study, women were given the option of knowing the facts about a potential date – his age, income, occupation – or knowing what another woman had made of him. Most wanted the facts, but the third-party judgment proved a better predictor of how the date would go.  Above all, Dolan urges, remember that happiness, unlike money, isn't very fungible: tolerating misery for years in the hope of a bigger happiness payoff later almost never works. Once a situation has been making you unhappy for a while, abandon it. Here he may be guilty of making happiness seem easier than it is. ("Find ways to laugh more" is one such piece of glib advice: if it were that easy, depression wouldn't exist.) But even if it isn't easy, Dolan makes a persuasive case that happiness might really be simple. His book is a powerful reminder not to get caught up in overthinking things, but to focus instead on maximising what actually delivers joy. "Listen more to your real feelings of happiness than to your reflections on how happy you think you are or ought to be," Dolan writes, and most of us would benefit from listening to him  Dolan also argues that it's "better for your happiness if you have modest expectations". If that's right, Vincent Deary's publishers are playing with fire in their campaign of pre-publication hype for How We Are, the first in a "monumental trilogy", immodestly entitled How to Live. (The others, already written and due in coming years, are How We Break and How We Mend.) This "mesmerising and universal portrait of the human condition", the debut of a 50-ish psychologist from Northumbria University, "has only recently come to light", hiding fully-formed in a desk drawer until Deary found the courage to retrieve it, triggering a multi-publisher bidding war. All of which makes How We Are sound dauntingly grandiose. In fact, it's a quieter achievement than that, and none the worse for it: a lyrical, consoling exploration of how tough it can be to start "swimming upstream" against "the prevailing forces of habit, inertia and gradual decay" – but how sometimes, when life summons us to change, we must answer.  Blending philosophy, psychology, personal anecdote, movies, literature and neuroscience, How We Are shows vividly how remaining stuck in our ruts – "saming", Deary calls it – isn't merely a failure to act, but something to which we subconsciously devote great effort. Like shoppers treading desire lines across the park to reach the supermarket faster, we embed shortcuts into life. We'd go mad if we didn't: our very sense of self is little more than this accretion of memories, habits and relationships. We mould our physical environments similarly, arranging furniture and files, coffee makers and computers to reflect our established patterns of thought and behaviour. "Give any of us long enough in a space," Deary writes, "and we will upload to it, imprinting large portions of our thinking and our feelings on to it."  The trouble comes when we need to fight this tremendous momentum of sameness, to move to a new phase of life: to start or end a relationship, move cities, become a parent, come out of the closet, switch jobs, confront bereavement, adjust to an empty nest, retire, or face death. We're like the new wife in Daphne du Maurier's Rebecca, Deary suggests, attempting to survive in a house in which every move we make has been prescribed by the habits and desires of someone now gone: "We live in rooms haunted by ourselves." No wonder that when we first hear the call to change, our first instinct is to put our fingers in our ears. Abandoning old paths for virgin soil is more than just hard; it's self-annihilating. But if we're to avoid "character sclerosis", we must do it, pushing ourselves through the awkward, unreal-seeming first days of a fresh way of life, until eventually the new becomes familiar, and automaticity takes over again.  This archetypal human story is brought to life in How We Are by means of an almost kleptomaniac approach to cultural references: we hear from Don Draper as well as Du Maurier, Daniel Dennett, Slavoj Žižek, Sigmund Freud, Terry Pratchett and the Buddha, George Eliot, Aristotle, President Josiah Bartlet, and participants in the 2006 season of Big Brother. (A curious absence is Carl Jung, whose ideas Deary's often recall.) Deary also gives the impression of having never encountered a movie – including even Ghostbusters, The Devil Wears Prada, 50 First Dates and Kevin Costner's Waterworld – that didn't somehow illustrate the challenge of outgrowing our routines. The cumulative effect of all this is mainly exhilarating, if occasionally exhausting. More importantly, Deary never falls into the trap of sounding as though he's got all the answers; his voice is the pleasingly sardonic one of a peer who finds this stuff as difficult and disorienting as the rest of us.  He concludes – until the next instalment, anyway – with a warning: it's far from inevitable that we'll find the will to break away from old routines, to achieve escape velocity from the pull of the automatic. Moreover, we can fool ourselves we're changing when we're not. (Travelling the world might jolt you out of a rut, writes Deary – "but I've met too many hardened travellers for whom travel is a habit, a way of staying asleep and avoiding the wakefulness that establishing a settled life would demand of them.") It takes guts to recognise that change is called for, and more to follow it through. This book – so long as you don't read it on autopilot – should help. And it will serve as a prophylactic against the alternative, to which it's dangerously easy to succumb: becoming a hardened shell of habits, with not much left inside.  • Oliver Burkeman's The Antidote: Happiness for People Who Can't Stand Positive Thinking is published by Canongate. To order Happiness by Design for £16 and How We Are for £13.59 with free UK p&p call Guardian book service on 0330 333 6846 or go to guardianbookshop.co.uk.
Stuck in a rut? Imprisoned by habit? Two major books offer insights into how to be happy and when to make a major life change, writes Oliver Burkeman