'In an instant, lightning flashes, and the burst might leave me blind. When the bolt of lightning crashes, and it burns right through my mind.' --from the musical 'Next to Normal,' lyrics by Brian Yorkey.  It's close to midnight in a Manhattan recording studio, and a songwriter is listening to a work in progress, one that he hopes millions will soon know by heart. He'd recently improvised the lyrics in the vocal booth, including the chorus: "If you're the country, I'm your new citizen." Now, he sings along with the playback in a soft, high voice. If all goes to plan, however, the voice forever linked to "Citizen" will be that of Beyoncé Knowles, the superstar singer in the next room, who's been gathering songs for a new album.  Known as The-Dream, the 32-year-old lyricist has cause for confidence. He already helped create one of Beyoncé's biggest hits, "Single Ladies (Put a Ring On It)." For much of last year, the track's stuttering beat was everywhere, made indelible by Beyoncé's riveting dance video and a bit of lingo The-Dream coined for her chorus: "If you like it then you should have put a ring on it." This cheeky rallying cry became something more than a pervasive hit: maybe a new standard.  Listen to the demo version of 'The House That Built Me' by song writer Tom Douglas and co-writer Allen Shamblin. In the demo, Mr. Douglas featured piano and strings, aiming for a cross between "Jackson Browne and Jimmy Webb."  When the song was recorded by country star Miranda Lambert, however, it became a hushed ballad featuring little more than acoustic guitars. Listen to a clip.  By The-Dream's count, that's one of at least two potentially timeless tracks under his belt, including Rihanna's "Umbrella." Key to his definition of a standard: the number of acts, major and minor, who adopt the song. "When Liza Minnelli sings 'Single Ladies,' I'm like"—finger snap—"got it," he says. "When Taylor Swift plays her acoustic version of 'Umbrella'"—snap—"got it."  Divorced from a song, words on a page look like poetry—usually bad poetry. "Lyrics just don't hold up without the music," says Billy Collins, professor and former poet laureate. When his students argue that the lines by their favorite rock stars should be assessed as literature, he demurs: "I assure them that Jim Morrison is not a poet in any sense of the word."  Without lyrics, the human voice would be just another instrument, yet the process of creating them is one of the most mysterious aspects of composition. There was a time when much of the nation knew "Stardust," by Hoagy Carmichael and Mitchell Parish, and could sing, "But that was long ago, now my consolation is in the stardust of a song."  Today, by its loose definition, the Great American Songbook is closed to new entries. The soul of popular music in America was shaped between roughly 1920 and 1960, primarily by Tin Pan Alley. Writers including Irving Berlin and Johnny Mercer cranked out songs that spanned Broadway shows, Hollywood movies, radio and records. "It was all jazz-based. Everything from the Gershwins to Hoagy Carmichael to Ellington—it all had the same tonal sound," says Deborah Grace Winer, artistic director of the Lyrics and Lyricists series at New York's 92nd Street Y.  That consensus began to erode in the rock and roll era. The musical universe has been dispersing into various genres and sub-genres, making it increasingly rare for songs to cross cultural lines. Most lyricists now labor in smaller cells, their words sung along to by niches.  A vestige of the old system lives on in Nashville, where Tom Douglas is a house writer for publisher Sony/ATV. Many days, he commutes 20 minutes to downtown in his Jeep Wrangler, often stopping off at Starbucks before a day of co-writing sessions at the renovated firehouse used by some 30 Sony/ATV writers. "There's a certain blue-collar work ethic to what we do," Mr. Douglas says.  A former real-estate salesman of 13 years, Mr. Douglas didn't get his first song recorded until age 41 with "Little Rock," which took the form of a phone call home from someone newly sober and starting over, "selling VCRs in Arkansas at a Wal-Mart." Released by Collin Raye in 1994, the song topped the country chart and launched Mr. Douglas's career. He's been under contract to Sony/ATV ever since, and had more than 100 songs recorded, including hits sung by Martina McBride, Lady Antebellum and Tim McGraw.  At the Sony/ATV writers' quarters, near portraits of writers such as Red Lane and Rodney Crowell, the sound of music leaks out of rooms that writers have booked to work in teams. One of the most recent releases with Mr. Douglas's name on it, "The House That Built Me," gestated for six years, starting when co-writer Allen Shamblin mentioned a line he'd read somewhere: "We don't build houses, they build us." The writers sketched out a song about someone visiting their childhood home, looking for direction in life. But it felt unfocused and the writers put it aside for four years until 2008, when the song gelled around a new refrain: "If I could just come in, I swear I'll leave, won't take nothing but a memory, from the house that built me."  —'Sorrow' by the National; lyrics by Matt Berninger  —'Single Ladies' by Beyoncé; co-written by The-Dream  —from the opera 'Ainadamar,' libretto by David Henry Hwang  The response from Sony/ATV's song pluggers, who play matchmaker between writers and recording artists, was immediate. "It was one of the five days a year where the flares go up and the beacons sound," Mr. Douglas says. The song went to country star Miranda Lambert. It was pushed as a single and spent four weeks at No. 1 on country radio.  On a recent weekday Mr. Douglas headed into a writing session with singer Shawn Camp, a Nashville fixture. He was armed with little more than three words: "Shatter the madness." He said, "Sounds like a title. I might lay that on him. It could be an up-tempo idea. 'Only love can shatter the madness.' "  With a loyal "Americana" fan base and a sound that weaves through country, folk and rock, Tift Merritt is working in the tradition of Joni Mitchell, James Taylor and Leonard Cohen. Singer-songwriters, pouring their lives into song, are still alive and well and on the road. But with the exception of a few commercially reliable balladeers like John Mayer—whose "Daughters" has become a wedding standard—few are creating crossover anthems.  Ms. Merritt says, "I can't explode the genre and reinvent it; I'm not that kind of genius. What I can bring is my own eyes and heart and hopefully there is a point of view, a voice."  On her fourth album, "See You On the Moon," Ms. Merritt inhabits a few characters, singing from the perspective of a juvenile offender and her own late grandfather. The song "All the Reasons We Don't Have to Fight" is the product of a clash with her husband (and drummer), Zeke Hutchins. Hung up on the poison that lingers after lovers quarrel, she wrote verses such as, "These words we shout, so who's to say, where they will go when they fall away? Maybe they hang around with the lonely kids, with a balled up fist, saying, you did this, you did this."  Some songs come in a torrent, she says, but she grappled with "All the Reasons" for five days, poring over papers taped to her windows. In process, some of her songs have filled entire notebooks (lately she prefers unlined, 120-page Moleskins with soft covers). The notion of ever being expected to write hits, or even to write with someone else in the room, "makes my stomach hurt," she says.  Songwriting is equally personal for indie rock star Matt Berninger of the National. With a stately, often melancholic sound that matches the singer's baritone, the band is currently topping major music festivals and consistently selling out theaters, even though its songs are virtually absent from commercial radio. It's the kind of shadow stardom playing out for icons of other subcultures, whose fans find a sensibility they strongly relate to. "I will admit that my lyrics often come from the perspective of a college-educated, middle-aged, anxiety-filled, average white guy, stressed-out by having a job, keeping a job and being a grown-up," says Mr. Berninger, lead singer and lyricist. "But we still have these fantasies of being brilliant and independent and reckless." Of his lyrics, which the National doesn't include with its CDs, Mr. Berninger says, "there aren't any messages or platforms for ideas, other than how it makes you feel."  Mario Quintero sings in Spanish to a part of America that is cordoned off by language and culture. The Spanish-language music market is immense, but also divided into subsets according to country of origin and style. Mr. Quintero is a leading star of norteño, a music that originated in northern Mexico with high-stepping tempos and accordion flourishes, bearing the influence of the polka brought to the region by 19th-century European immigrants. Mr. Quintero's lyrics, however, reflect some controversial 21st-century themes, including immigration and drug trafficking.  The lyrics to his "Los Ilegales" translate as, "We are wetbacks, this is true, but it isn't bad to be an illegal, we are humans just like you." The song is also defiant: "That big wall doesn't stop us, we will cross and by multitudes. And even if you do not give us a driver's license, we will travel by the grace of God."  Mr. Quintero shot to fame in the 1990s, thanks in large part to his output of "narco-corridos," fables that detail the exploits of real-life drug kingpins. Most narco-corridos have been banned from Mexican and U.S. radio, but Mr. Quintero defends the music. "I'm not saying what [the drug lords] do is right, but I give the audience entertainment. It's like watching an action film. Sometimes you want to see the bad guys getting away with it," he says through a translator.  For a writer working part-time in the "sort of 19th-century profession" of opera librettist, David Henry Hwang has been on something of a hot streak. According to Opera America, four operas that he's penned lyrics to, including "Ainadamar," have been performed by various companies in the last five years, more than any other living librettist during that time. Not that he has much competition: As many opera companies try to build audiences with new twists on the classics, investing in new repertoire represents "one of the most daring things you can do in the music world," says Fran Richards, who oversees operatic works for performing rights organization ASCAP.  Primarily a playwright, Mr. Hwang has collaborated on four operas with composer Philip Glass. He typically drafts a libretto before hearing the music. Guided by discussions with a composer, whose artistic vision he defers to, Mr. Hwang writes verses on a legal pad to the "crappy melody" playing in his own head. He paces his lines to account for a singer's long notes, and leaves lines open-ended with vowels.  Mr. Hwang's operatic retelling of David Cronenberg's 1986 horror movie "The Fly," composed by Howard Shore, featured a teleportation device, a shotgun killing and lines from the half-man/half-insect protagonist such as "I'm sick of your fears, sick of your caution. It's best for you, best for me, the next step in evolution." Says Mr. Hwang: "It had all those great opera elements of sex and death."  If Nashville carries on the work ethic of Tin Pan Alley, many of its stylistic torchbearers can be found on Broadway. "There's a conversational ease and a measure of wit that you aspire to. You probably never reach that level, but that's the gold standard," says lyricist Brian Yorkey, citing the likes of Cole Porter, whose rhymes included, "Good authors too who once knew better words, now only use four-letter words writing prose, anything goes."  Mr. Yorkey, working with composer Tom Kitt, used rock music to create Tony Award and Pulitzer Prize winner "Next to Normal," a stormy yet funny portrait of a family dealing with mental illness. He and other writers are testing one of the main tenets of musical theater, that lyrics must adhere to exact rhyme schemes. Classic example: "Oklahoma, where the wind comes sweepin' down the plain, and the wavin' wheat can sure smell sweet, when the wind comes right behind the rain." Ted Sperling, a music director who helped bring the latest version of "South Pacific" to Broadway and who oversees the development of new musicals at the Public Theater in New York, says, "Writers now think that's old-fashioned. My sense of purity bristles at that, but I can understand."  "Next to Normal" has its share of rollicking show tunes. But in swaggering numbers such as "I'm Alive," looser rhymes ("memory" with "mystery") are part of a pop vocabulary. Says Mr. Yorkey: "A rock hook is different than a 32-bar Cole Porter song that spins out into something transcendent. Rock music stamps its foot on the ground and sings 'I'm alive!' four times in a row."  Oddly, the musical genre that in many ways resembles Broadway in the lyrical discipline it requires is hip-hop. For the "Anthology of Rap," to be published this fall, author Adam Bradley compiled about 300 songs that helped make the music unique among other genres: "For three-plus decades of rap, the lyric has always been at the center," he says. He chose milestones of style and concept, including the portrait of psychosis in Eminem's "Stan," Brother Ali's scenes of family struggle in "Room With a View," and the assonance and alliteration in UGK's "Pocket Full of Stones." Mr. Collins, the former laureate, says he admires the poetic demands of the form. "The rhymes need to be relentless and they need to be clever. Hip-hop is a great home for the human love of rhyme."  When The-Dream (born Terius Nash) came up with the "put a ring on it" hook to "Single Ladies," he told himself, "All I have to do is write around that. I'd have to be a fool to f— this up." It was 2008, and he was in the studio with Beyoncé for the first time. Adding to the pressure, another team of "very big guys" was working in an adjacent room, competing to land tracks with the singer.  As his production partner, Christopher "Tricky" Stewart, tinkered with a skeletal beat, an urge spurred The-Dream to the microphone. He hummed the cadence and melody, then filled in the words to fit the attitude of the woman he was imagining: "I know that girl, the one who's not quite over you but says she is." He says the song was virtually complete in 20 minutes.  Write to John Jurgensen at john.jurgensen@wsj.com
Modern lyricists are fighting to keep their words in tune with a wildly changing music business. Some of the best of them talk about how they nail the phrases they hope will last forever.