Watching Walking on the Wall, a 1971 work by US choreo-grapher Trisha Brown, is an extraordinary experience. The dancers, held parallel to the floor by unobtrusive harnesses that slide on tracks, almost glide. Spurning gravity, they never look down as they put one foot in front of another, step around corners, meet and part. Seen from the balcony of the Barbican, it's like watching life on the street: people strolling alone and in pairs, holding hands. Then you notice the other spectators in the well of the space below, looking up. But which way is up? You feel like a character in an MC Escher drawing, caught up in an optical game with impossible architecture.  Three Brown works from the 1970s are performed every hour, throughout the day, in Pioneers of the Downtown Scene, a look at art and life in New York's SoHo during the 1970s. Back in the day, Brown had dancers performing on the flat roofs, fire-escape ladders and water towers of Manhattan. In one mesmerising photograph, your eye discovers figure after figure performing across the skyline. Brown also had roped-up dancers walking down the sides of buildings, and, in Floor of the Forest, cradled in mid-air in a nest of old clothes and rope. They rummage amid the jumble and hang upside-down, limbs poking out everywhere.  It was a time when art was moving off the plinth, and dance was stepping away from the proscenium stage: the two joined forces and invaded the galleries, museums and lofts of New York. Brown, turning walls into sidewalks, made a stage of the city itself.  New York in the 1970s was broke, busted and frequently dangerous. Even at the end of the decade, I was advised to carry mugging money, to walk under the streetlights, and never to stroll through Central Park at night. It would have been safer to walk on the walls. Forget Paris. New York, especially SoHo, was where aspiring British artists yearned to be. Britain's own DIY artistic culture in London's East End paralleled New York's, but with less elan and almost no audience.  At the end of the 1960s, SoHo was a district of abandoned workshops, moribund sweatshops and acres of empty floorspace. It began to be populated by artists who had little money, few prospects, and lived and worked there illegally, at first. Loft-style living is now an estate-agent cliche, but in 1970s SoHo it was the real deal. Today, the area is just chi-chi. The artists moved in, then the galleries, then the money. Pioneering artists are now moving to Detroit, the rust-belt city whose stunning, ruined architecture can be had for a song. They're building communities there, starting again. Who knows where this might lead?  Pioneers of the Downtown Scene focuses on the social world, the experiments and the art of Brown, Laurie Anderson and Gordon Matta-Clark, son of Chilean surrealist painter Roberto Matta. Phillip Glass was there, Steve Reich, Joan Jonas, Yvonne Rainer and legions more. They fixed plumbing and built walls; they botched electrics, made do and got by. Artists got into dance; dancers and musicians got into their art. They were each other's influence and audience, a clique not yet riven by commerce and fame.  Feminism, protests against the Vietnam war, a social conscience and communal spirit were more than a backdrop. Anderson started out drawing political cartoons and being an "art reporter". She also photographed strangers who had assaulted her with sexual remarks in the street. She recorded the dreams she had after falling asleep in incongruous public places, and played a violin weirdly adapted with shoddy electronics from Canal Street markets. She told laconic stories as unhinged as the times. Her later success as a multimedia artist was formed in these works in downtown New York. It's fascinating and a little ramshackle, and all the better for that.  Matta-Clark opened a restaurant in SoHo called Food, to feed the burgeoning artistic population. He also covered himself in shaving foam while roped to a huge clockface high above the Manhattan street, behaving more like silent movie prankster Harold Lloyd than an artist. He devised dances in trees, imagined floating islands in the Hudson, and started slicing up buildings, excavating floors and hacking through walls in the Bronx. Like Brown, Matta-Clark defied both gravity and architecture, and probably sense, too. He died of cancer in 1978, aged 35.  The art, music, dance and theatre that came out of the downtown scene was born of contingency. This show effects a well-tuned balance between art and archival material, photographs, drawings, performance and grainy old film and video. There are bits of Matta-Clark's dissected buildings and a big waste container fitted out with internal walls and reclaimed doors – turning it into a house with many rooms, prefiguring the installations Mike Nelson makes today. The exhibition charts a social as much as an artistic moment, with its friendships and love affairs, ideals and arguments. There was little distinction between work and play. It was a prolonged, vital moment, and the work still resonates. This exhibition is worth the trip for Trisha Brown alone.  Every time I went to New York from the end of the 1970s till her death in 2009, I would try to visit the artist Nancy Spero. She lived with her partner, the painter Leon Golub, in a big loft with a dividing wall between their studios, on the border of SoHo. "You're a marked man in this town," Golub once said, threateningly, when I began writing criticism. Golub affected to be a Chicago bruiser, and painted with a meat cleaver. Spero seemed a gentler soul, till you saw the rage in her art. Every time I met them, I felt chastened by a somehow old-fashioned yet relevant bohemian morality and ethical certainty. Spero was already a legend, a key figure in the feminist movement, an outspoken critic of injustice.  A small, measured retrospective of Spero's work is currently at the Serpentine in London, and a new book by Christopher Lyon documents her entire career. The Serpentine is a good venue for her art, with its disconcerting mix of intimacy, anger and celebration.  For many years, Spero suffered from arthritis: hand-painting gave way to stencilling and home-made printing. She developed a cast of characters and motifs that populated frieze-like drawings, or were hung – gurning and snarling – from a maypole, which now fills the Serpentine's south gallery. You need to approach her art with care. Suddenly, you're jolted by newspaper clippings of horrors perpetrated in El Salvador and the Philippines; then goddesses, phallic women and neolithic fertility gods dance and sing and bare their vaginas at us.  The verbal assaults of French poet, theatre director and madman Antonin Artaud rage across her drawings. Her work erupts, joyously and horrifically, as it erupted year after year in Manhattan, impervious to fashion, unfazed by the growing and sometimes overnight successes of neighbours and colleagues, the calumnies of US politics, domestic and abroad. She watched like an eagle. In her art, there are moments of sexual frenzy and abandon; then the helicopters come, shitting death. SoHo may be over, but idealism isn't.  • Pioneers of the Downtown Scene is at the Barbican, London EC2, until 22 May. Nancy Spero is at the Serpentine, London W2, until 2 May.
In the 1970s, Adrian Searle used to stroll through downtown New York carrying money for muggers, on his way to meet a man who painted with a meat cleaver. What does he make of a new show about the era's art scene?