The physics lab that brought you the Web is reinventing the Internet. Get ready for the atom-smashing, supercomputing, 5-gigabits-per-second Grid Economy.
200 feet underground, a proton does 17-mile laps at nearly the speed of light. Guided by powerful magnets, it zooms through a narrow, circular tunnel that straddles the Switzerland-France border. Then a tiny adjustment in the magnetic field throws the proton into the path of another particle beam traveling just as fast in the opposite direction. Everything goes kerflooey.
This article has been reproduced in a new format and may be missing content or contain faulty links. Contact email@example.com to report an issue.
This will happen 10 million times a second inside the Atlas detector, part of the Large Hadron Collider now under construction at CERN, the famed European particle physics lab outside Geneva. When the LHC is finished in 2007, it will be the largest accelerator in the world. Massive superconducting magnets cooled to near absolute zero by liquid helium will bend 20 micron-wide beams of protons into precise trajectories and crash them into each other.
Photo by Maximilien Brice/�CERN
The Large Hadron Collider, under construction at CERN, is targeting the elusive Higgs boson. But crashing protons is the easy part.
Hadrons are a class of subatomic particles that includes protons and neutrons. When they collide, they explode into dozens of other particles, even more infinitesimal and fleeting. Atlas, five stories high and one of the most complex experimental apparatuses ever built, is designed to see them all.
The cost: $3 billion and change. The goal: to find one lousy subatomic particle.
Specifically, the Higgs boson, the most elusive speck of matter in the universe. Often called the God particle, it's supposed to be the key to explaining why matter has mass. Physicists believe that Higgs particles generate a kind of soupy ether through which other particles move, picking up drag that translates into mass on the macroscopic scale. The Higgs is the cornerstone of 21st-century physics; it simply has to be there, otherwise the standard model of the universe collapses.
For all the high-level physics, smashing protons together is actually the easy part. The hard part is crunching data. To find the Higgs, which might flash across Atlas' layered detectors for a microsecond, researchers will have to process a staggering amount of information. Atlas and its three sister detectors will spew a thousand times more raw data in a year than in all the world's phone calls. Every eight-hour run of the LHC will produce around 10 terabytes. At full power, the LHC could produce 10 petabytes of useful data each year. That's 1016 bytes – 2 million DVDs' worth of binary numbers encoding energy levels, momentum, charge – all in search of the one in 10 trillion anomalies that could mark the passage of a Higgs.
Discovering the Higgs might seem an esoteric goal. But the search will have a powerful real-world spinoff: to process all that data, scientists are building a worldwide meta-network of PCs, organized into large clusters and linked by ultra high-speed connections into a global, virtual computing service. It's called the LHC Computing Grid, and it could mark the evolution of the Internet from a pervasive communications network into a powerful, global computation network.
The LHC grid is already lighting up – a major test is scheduled for April. There have been other grids, but none with the scale, complexity, and power of the one they're building at the LHC. Many hurdles remain, but large corporations, investment firms, and the kinds of people who work with big loads of data have already glimpsed the possibilities. If the LHC grid works, remote supercomputing may become as common as electricity, as ubiquitous as the Web.
An undergrad at the University of Edinburgh in the 1960s, Les Robertson studied with Peter Higgs – yes, that Higgs. In 1974, Robertson came to CERN's computing department. Today he's the head of the LHC grid. When the Large Hadron Collider was approved in 1994, Robertson – a pragmatist at a lab full of woolly-headed theorists – realized that no supercomputer in the world could handle the massive amount of data the new collider would produce. And he knew he'd never get the funding for a supercomputing center anyway.
The answer, Robertson figured, was to link together thousands of smaller computers. "I had assumed we'd do this in a simple way," he says. "We'd develop the management software ourselves and pass bits of data down an inverted tree" – in other words, data would flow to nearby centers, and results would come back in a hierarchical, predictable way. The centers wouldn't be linked into any kind of virtual supercomputer; they wouldn't have to be.
Then, in the summer of 2000, a little-known computer scientist named Carl Kesselman came to Geneva from California to give a seminar. Kesselman is one of the fathers of grid computing; Robertson had heard of the concept, but now he was seeing the light. A grid was the perfect solution for the enormous amounts of processing capacity the LHC would require -�and it was dynamic, flexible, and infinitely expandable.
To understand why, think of SETI@home. Individual PC users donate spare CPU cycles to analyze the apparently random radio signals constantly bombarding Earth from deep space. In return, they get to participate in a grand scientific quest to find evidence of life Out There. No luck yet, so we're told, but the concept has worked beautifully: It has proved that computing power even at the scale of the desktop can be mobilized remotely.
In a similar way, the LHC grid will tap computing resources from around the globe. The first iteration, in 2006, will involve some 200 scientists from 150 institutions in three dozen countries. It's not exactly like SETI@home; users will actually have access to all that computing power and be able to do their own individual analyses. However, computing centers and high-energy physics labs don't have "spare cycles." So the trick will be allocating powerful but finite resources in an efficient and managed way. The nodes of the LHC grid – mostly universities and government-supported research labs – will agree to supply a share of their computing resources, in return for LHC collision data they need to help find the Higgs. Those agreements will vary between nodes, depending on which experiment a local physics group is collaborating on. Eventually, according to Harvey Newman, a physicist at Caltech and chief architect of the US-based portion of the LHC grid, these agreements will morph into a genuine "grid economy," wherein computation, storage, and network resources will be traded around the grid like soybeans in a commodities market.
The result: a super-reliable, superpowerful network that supplies on-demand computing capacity anytime, anywhere.
The key is middleware called the Globus Toolkit, an open source project led by Kesselman (who now runs the Information Sciences Institute at USC) – and Ian Foster of the Argonne National Lab in Illinois. They're the Lewis and Clark of grid computing – in fact, they wrote the 750-page book on it. Globus enables a grid to interpret a user request and then autonomously find the appropriate computing resources. Then it breaks the job into the right kinds of smaller tasks, allocates the spare computing power, and gets to work solving the problem. Robertson and his team at CERN have also integrated software developed for various grid projects around the world, but the Globus Toolkit still provides core protocols.
On the LHC grid, that software is supposed to solve a colossal signal-to-noise problem. It involves combing through mundane particle interactions for traces of "new physics" and comparing those traces to a simulation of what Higgs is supposed to look like. The CPU cycles could come from Birmingham or Berkeley or Beijing; a physicist working the problem won't know where the capacity comes from, and won't care.
In the 1960s, networking pioneers like J.C.R. Licklider and Leonard Kleinrock envisioned the spread of what Kleinrock called "computer utilities" – pervasive services that would provide remote computing to individuals. In the 1990s, the spread of computer clusters and the business model known as Web services brought that vision closer to reality. But clusters are by definition centralized, independent nodes of computers managed by a single authority. It took high-speed networks and sophisticated software – much of it created by Foster and Kesselman – to make grid computing possible.
"We're still doing things the old-fashioned way: You carry around your own computer, that's your computation tool. The grid will change all that," says Foster. At least, that's the theory. To find out if it works, its architects are pumping more and more simulated data through the grid, waiting for the Atlas detector to come online.
When I toured CERN last fall, Atlas was under construction on the back lot in a building as big as a hangar. The components are so heavy that they rest on supports that float on cushions of pressurized air, as precarious as elephants on skateboards. ("You could push one of these with your hands," physicist Robert McPherson told me as he showed me around. "But you don't want to.")
Later, I stood in the cathedral-sized cavern that will eventually house Atlas, 200 feet beneath the Swiss countryside. Jean-Luc Baldy, CERN's head of engineering, gestured toward metal platforms that stretched hundreds of feet above us. "The easy part was digging the hole," he remarked with Gallic understatement. "Getting the machine into the hole is much harder." As we talked, powerful forklifts moved blue steel beams as welders built a cradle for the detector, which will weigh as much as a small ocean liner. Above Baldy's head, laborers in hard hats clambered over the metal scaffolding, finishing the chamber's interior. It was like a scene from Dr. No; at any moment I expected a nuclear-tipped missile to rise from the floor and the roof to slide open.
As particles arc through Atlas' intricate viscera after a collision, they leave traces, electrical impulses that Atlas translates into streams of 1s and 0s. Transmitted to physicists' desktops around the globe, those numerical streams may contain the evidence for the God particle.
The problem here is volume, not complexity. To model the chaotic effects of, say, earthquakes or the global climate, you need complicated algorithms and intelligent software; by contrast, the work of the LHC grid is essentially data mining in a huge mountain of numbers. As Robertson puts it, the output from the LHC is "perfectly parallel." Each collision is an independent event, separate from the previous one and the next, so it's relatively easy to break the datastream into bundles and analyze each in parallel, across many distributed computers, before recombining them. From their mathematical models, physicists can deduce the particular combination of energy, momentum, and charge that the Higgs boson should have. Teasing it out from all the other, mundane particle tracks is "a big job, requiring vast amounts of computing power," Robertson says. "But you can just break it into as many jobs as you have processors available, run it as long as it takes, and then put the results together."
Will the LHC grid perform as they hope? The networking speed is already there. Last October, Newman's group at Caltech and Robertson's team at CERN set a new Internet land-speed record for data transfer. They sent a terabyte of data across 4,400 miles in less than 30 minutes. That's 5.44 gigabits a second, or about one DVD every seven seconds – five times as fast as the old record, which they themselves had set just eight months earlier.
Network infrastructure also poses no problem. Thanks to the excesses of the 1990s telecommunications bubble, there's plenty of so-called dark fiber in the ground, laid but unused and going for cheap. So grid projects like the LHC can afford to lease their own networks rather than pay for bandwidth.
The real obstacle is cost. Serving just 100 scientists a day – 100 workstations operating on a terabyte of data – would fully occupy a 10-gigabit link, the largest pipeline in production today, according to Newman. While that's cheaper today than it once was, it's still a bill for someone. And that doesn't include the cost of the computational power at the centers that will contribute to the grid.
"The original grid concepts implicitly assumed that there would be plenty of resources," says Newman. Plan it, figure out how it will work, and grid computing should just happen. But "in the real world, it doesn't work like that."
That's the crucial distinction between grids and the Web. Information may want to be free, but doing work with it costs money. "When we make this leap to utility processing, it will theoretically give you access to computing power without knowing the details, just as the Web gives you access to information without knowing where it's stored," says Robertson. "But on the Web, there's so much information available, because people want to make it available without charging for it, and the cost of making it available is quite low. But is there so much high-performance computing capacity available that people want to give it away?"
Maybe not, but corporations might be willing to pay for it. Already IBM, Hewlett-Packard, and Sun have large and well-funded grid computing programs under way. The potential for an f/x designer, a civil engineer, or a statistician to tap into remote computing power gives CEOs visions of undreamed-of efficiencies – and will continue to drive investment in grid computing's future.
Then there's you and me. Much of the early hype around grids has spoken of "the Grid" as if there will be one pervasive ultra-network, the way there's one Web. After all, it's being birthed at CERN, where Tim Berners-Lee invented the Web itself more than a decade ago. It's easy to understand why people might still look toward Geneva for radical reimaginings of the ways we use computers. But builders like Robertson and Newman cast a skeptical eye on some of the wilder claims about the magic of grids. Not everything lends itself to parallel processing. Existing grid applications are mostly in the sciences (see "Grids Are Not Vaporware," left). Others might come from arenas that rely on data-intensive calculations: drug design, car crash modeling, cooperative film editing, finance, compression and expansion of massive content files – maybe even optimizing the distribution of goods over wide areas (solving the famous Traveling Salesman problem for actual traveling salesmen). In other words, real concerns, but not exactly everyday ones.
But then, the Web wasn't originally intended for civilians, either. People have a way of repurposing powerful technologies in surprising ways.
Meanwhile, the creators of the LHC grid have a "data challenge" in April to test the new grid with a massive flow of simulated data. It's nerve-racking – their grid still has a way of crashing at inopportune times. It remains a primitive work in progress. Just like the Internet before Tim Berners-Lee first wrote www.
The LHC Computing Grid
Proton collision data from the Large Hadron Collider is collected and crunched by CERN's supercomputing center.
Data from CERN goes to other major research centers, which divide up the analysis.
Computer centers elsewhere – labs and universities – also contribute computational resources.
Collaborators around the world connect to the network and add even more computers.
The end points; researchers get Atlas data, ultrafast connections, and powerful processing.
Grids are not vaporware
The CERN project will join a handful of computing grids that are already operational. Here's a look at some others:
Provides 4.5 teraflops of computing power for collaborations on dark matter, real-time weather prediction, molecular assembly, and other research.
Sponsors: National Science Foundation, various US labs and computing centers
Gives remote access to a steerable, high-energy electron microscope at UC San Diego, as well as data and sims dealing with structures at nano and micro scales.
Sponsor: National Center for Microscopy and Imaging Research
Uses the CPU power of investors' PCs to analyze stock market data.
Sponsor: i42 Informationsmanagement
Enables engineers and geologists to collaborate on 3-D, real-time earthquake simulations to design safer buildings.
Sponsors: Network for Earthquake Engineering Simulation, National Center for Supercomputing
Applications, National Science Foundation
Biomedical Informatics Research Network
Collects and analyzes magnetic resonance images of the brain for researchers studying Alzheimer's disease, depression, and schizophrenia.
Sponsor: National Institutes of Health