When Google was building Google Earth and Google Mars a half decade ago, Chris Kemp was amazed at how easily the company could pump up these massive web services with additional computing power.
"With Google, infrastructure was treated as if it was infinite," remembers Kemp, the former chief technology officer at NASA, which worked hand-in-hand with the search giant on Google Earth and Google Mars.
As a means of powering all its web services – from Google Search to Gmail to Google Maps, Google Earth, and Google Mars – Google spent years building what amounts to a single software platform that spans a worldwide network of data centers. The company calls it "warehouse-scale computing." The idea is to treat an entire data center as a single warehouse-sized computer that can juice an application with additional processing power, storage, and other resources whenever they're needed.
In recent years, Google's distributed infrastructure has inspired much of the tech world to move in a similar direction. Chris Kemp helped found the OpenStack project, an effort to bring Google-like flexibility to data centers everywhere. Yahoo and Facebook bootstrapped Hadoop, an open source number-crunching platform based on software that underpins Google's services. And several companies – including Amazon, Microsoft, and Rackspace – have introduced "cloud services" that offer the world this sort of scalable infrastructure via the internet.
But Google isn't content to merely inspire. It now wants to compete with the Amazons and the Microsofts. In the spring of 2008, the company unveiled Google App Engine, a service that lets anyone build and host applications on its internal infrastructure, and this week, at its annual developer conference in San Francisco, the company is expected to revamp this service in an effort win back some mind share – and market share – from Amazon, whose cloud services now run as much as one percent of the internet.
When first launched, App Engine wasn't all that different from the infrastructure services Google offered to its own engineers. This meant you could build an online application that would automatically scale to more and more servers as it was used by more and more people.
"App Engine came out of the philosophy of internal development at Google," Google product manager Greg D'alesandre once told us. "You don't want to have to worry about scale once scale hits you, because then it's too late. At Google, it's something we always plan for. You don't rewrite your application once it starts to scale. You write it so it can scale up front."
But there was a price. In building your application, you had to follow a strict Google rulebook. All applications had to be built with certain programming languages, and even within these languages, there were restrictions on the libraries and frameworks you could use and the way you handled data and the amount of time parts of your application were allowed to run.
In short, Google made things easy once your application was up and running, but you had to jump through a few additional hoops to get there.
These same kinds of restrictions had long applied to Google's internal engineers and applications, but with App Engine, they seemed to hold back its progress. Unlike Google's engineers, the average developer wanted to build applications in more traditional ways. And that meant he wanted to build applications on Amazon, a service didn't include the same restrictions as App Engine.
With its Elastic Compute Cloud service, or EC2, Amazon departed slightly from the Google model, offering access to raw virtual machines that let you run almost whatever you wanted to. Unlike App Engine – which hid all raw infrastructure – you had to juggle these virtual machines on your own, but the average coder still preferred the extra freedom.
Now, four years on, it appears that Google will revamp its cloud service, offering access to Amazon-like raw virtual machines as well. Gigaom reports that the company is "likely" to introduce such a service at its Google I/O developer conference later this week, citing multiple unnamed sources. And Google is now pitching App Engine and other services under a new moniker: the "Google Cloud Platform."
Google declined to comment on the Gigaom story. But the move is a long time coming. Over the years, Google has gradually reduced the coding restrictions on App Engine in an effort to reach a wider audience. Originally, for instance, you had no choice but to format your data for use with Google's in-house database, a "NoSQL" distributed database known as BigTable, but the company now offers a traditional relational database as well.
In May 2010, Google introduced a sister service for storing massive amounts of raw files. Known as Google Cloud Storage, this is akin to Amazon's Simple Storage Service, or S3, and now, Google is rolling out a service that's similar to Amazon EC2. Or at least it looks that way.
In some ways, this seems like another comedown for the web giant. Google has long touted that benefits of its "platform cloud" model, as opposed to Amazon's "infrastructure cloud." But as with the company's recently revamped Chromebook laptop, this shows that Google is determined to compete for the hearts and minds of businesses – and actually make some money along the way.
Microsoft is equally determined to compete in this burgeoning market. Recently, it too overhauled its cloud service so that it offers raw virtual machines. Google may have pioneered the notion of the infinite infrastructure. But when it comes to commercial services, everyone is chasing Amazon.