As PCMag lab analysts, our job is to take technology products through the paces and report our experiences with them back to our readers. No matter how a product fares, there is always a healthy respect for the sweat and ingenuity that developers, engineers and computer scientists put into the products that we test. So, after watching IBM's super computer Watson, make its official public debut on the quiz show, Jeopardy!; I marveled at the prowess of IBM scientists for creating such an awe-inspiring machine, but couldn't help noting a few areas where Watson came up short and wondered how it would rate in our lab.
Watson's most obvious gaffe was its inability to recognize when an answer had already been given. One of two of the all-time Jeopardy! champions pitted against the super-machine, Ken Jennings, answered a question incorrectly with, "What is 1920s." Watson repeated the same answer. Steve Camepa, IBM's general manager of global media and the entertainment industry explained that "Watson only takes his input from the question board so the fact that somebody else gave the same answer doesn't factor into what Watson says. He can't hear what the other players are saying, but maybe that's a feature we can add in the future."
In other words, a feature that is on the roadmap something we analysts hear quite often from vendors! It seems, considering the very nature of the Jeopardy! game, the ability to recognize what other players answered would be a necessary capability.
Watson was marvelous at coming up with "cut and dried" answers: questions pertaining to song lyrics or historical facts. It seemed to falter though at those "nuanced" questions so prevalent in Jeopardy!, the ones where the answer takes a bit of creative thinking and is often not so apparent. Take for instance one question, "From the Latin for end, this is where trains can also originate." The correct answer, "terminus" was given by Jennings. Watson, gave the incorrect answer for the question, but technically got the part right about "From Latin for end" with its answer, "finis." It's these types of subtleties that Watson was unable to grasp.
Watson, in general came across as a self-contained search engine. As host Alex Trebek pointed out, Watson is not connected to the Internet. The questions that it did best at are ones that if you entered into Google or Bing, you can get the same answers. For instance, if you input one of the questions asked in the Jeopardy! tournament into Google, "Bang, bang, his silver hammer came down upon her head" one of the first results is "Maxwell's Silver Hammer" which Watson correctly answered. You get the same results with Bing. It's as if Watson is using the same sort of search algorithms, except not culled from the Internet, but a manually compiled, ginormous database of song lyrics, history, literature and other concrete, indisputable bits of information.
It's still an exciting display of Artificial Intelligence and leads to more questions: can Watson learn so-called nuanced questions the more it is used? Can the algorithms "evolve" to mimic human thought?
If all of that isn't enough to arouse nerd sensibilities, consider Watson's amazing specifications:
- 90 IBM Power 750 servers enclosed in 10 racks
- 16 Terabytes of memory
- A 2,880 processor core
- Linux system
- While not officially disclosed by IBM, Watson is estimated to have cost $1-$2 billion
- Uses "DeepQA": a technology that enables computer systems to directly and precisely answer natural language questions over an open and broad range of knowledge
So, how would I rate Watson based on PCMag's tech product scoring system? For the first of a three-night performance evaluation a very good 3.5 stars with some points off for not being able to "hear" other contestants and not quite as adept at "nuanced" questions. After all, Watson is tied with human competitor Brad Rutter and Ken Jennings is not too far behind in the game. The fairest way to rate though, is to score each night individually; then average that total score and of course, to continue to be amazed at this intriguing technology from IBM.