Today at IBM's headquarters in Yorktown, New York, an historic battle was staged. Two superstar Jeopardy! alums (Ken Jennings and Brad Rutter) faced off against IBM's supercomputer Watson in a preview round of America's most challenging trivia game, and we were there to see the thrilling man-on-machine action first-hand.
Watson, named after IBM's founder, is one epic supercomputer. To handle the formidable task that competing on Jeopardy! presents, IBM spent years constructing a computer with 2,800 Power7 cores. That power is absolutely necessary--a single-core CPU, like in many modern computers, takes about two hours to come up with an answer to a standard Jeopardy! question, rather than the three-second average Watson currently boasts.
A lot of the challenge in creating an algorithm that can answer Jeopardy! questions lies in the questions themselves--the language used in these questions is hardly ever simple, often incorporating wordplay, riddles, and irony--but there's an additional problem in the addition of risk. In a split-second, a competitor must assess confidence in the question, weigh that confidence against the penalty of getting it wrong, and decide if the question is worth answering based on those factors. That's an intuitive effort for a human, but Watson had to be programmed with some incredibly complex reasoning to be able to do the same thing.
Watson has a certain self-awareness; it knows it won't get every answer right, and has to pass a certain level of confidence before it will answer. Watson's logo will change color to indicate its confidence: The lines that are part of its "avatar" will glow blue if Watson is confident, and orange if it's not.
The vagaries of language mean that the questions can be interpreted in all kinds of different ways, so merely figuring out what the question is trying to ask provides the majority of the struggle for Watson. To that end, the computer actually comes up with thousands of different possible answers, and ranks them by the possibility of correctness. When we watched the quick match, the top three answers were displayed on screen, as well as the confidence percentage, and the second- and third-ranked answers were usually dramatically incorrect. It's not likely that Watson will confuse, say, the author of one children's book with the author of another. It's more likely that Watson will completely misread what the question is even asking, and come up with an answer like "What is children?"
In this introductory battle, we learned a few things about the adjustments made to the show to accommodate a more mechanical being than usual. The question feed goes directly into Watson, so it doesn't have to "read" the question like the human competitors. But Watson does have to press a physical button to ring in, just like Ken Jennings and Brad Rutter, which pretty much eliminates the split-second advantage the computer has.
Interestingly, Watson will not be connected to the Internet, so there won't be any instant Wikipedia lookups. (IBM's reasoning: "Ken [Jennings] and Brad [Rutter] aren't connected to the Internet, so Watson shouldn't be either.") So where does this AI brain get its information? IBM's engineers, without the benefit of the Internet, have to load all of Watson's information manually, which includes encyclopedias, thesauruses, dictionaries, books, screenplays, and other compendiums of human knowledge.
There will be no audio or video clues in the eventual game, though the questions that require betting--Daily Double and Final Jeopardy--will remain. Watson performs a risk analysis on the categories given for those types of questions, though his precise reasoning means that his wagers are often unusual figures (a human might bet $2,000 instinctively, but Watson's risk assessment might indicate that a bet of $1,986 is more prudent). Watson actually learns in real-time, within the category--if it doesn't immediately understand a category, it will wait until a question or two in that category has been asked, and then use that data to figure out the pattern. Watson also takes the competition into account: If it's losing, it might adjust to answer questions with which it has less confidence than if it was sitting on a large lead.
I spoke to David Ferrucci, the Principal Investigator for Watson's DeepQA Technology at IBM, about the things Watson struggles with. "The things that are most difficult for Watson," he said, "are the things that haven't been written about." Small items that may stick in a person's mind that could lead to the answers of trivia questions are not nearly as accessible to artificial intelligence programs like Watson, even with its massive memory bank.
Certain elements of human language are tricky, too--the stuff that seems like it might be the most difficult (like puns and wordplay) are felt out by "trigger" words in the category name, such as "sounds like." But synonyms are often a bigger problem. In the answer "This liquid cushions the brain from injury," Watson has to determine that "liquid" is in this one case interchangeable with "fluid," and that "cushions" is interchangeable with "surrounds." Humans know what the question is asking instinctively, but Watson has to analyze it from every angle.
In the preview match I saw, which was all too quick, Watson performed surprisingly well. Not just well; it won handily, with $4,400 to Ken Jenning's $3,400 and Brad Rutter's $1,200. None of the contestants, human or machine, actually got a question wrong, but Watson seemed to be fastest at chiming in. Its weakest category was "Children's Book Titles"; Ken Jennings nearly ran the category, and Brad Rutter later quipped that "Neither Watson nor I have kids."
The eventual contest will be a two-day tournament format, in which the competitor with the most amount of money after two days will be crowned the victor. The winner will be awarded $1 million, second place $300,000, and third place $200,000. IBM will donate the entirety of Watson's winnings to charity, while Ken Jennings and Brad Rutter will donate half of their winnings.
Who will emerge victorious with such a big purse on the line? Will flesh and blood greed add caginess to the human competitors, giving Watson an advantage? What anecdote will Watson produce during Trebek's condescending interview segment? (Trebek says he'll "probably try to have a little fun with him.") We'll have to wait until February 14 to find out.
how much would it cost to get one of these to take tests for me XD
so...anyone else on edge about all this ai partial self-awareness stuff?
As long as they don't give it weapons or legs, i feel pretty safe.
If Watson misses an answer look for one of the contestants to say, "It's elementary my dear Watson."
@i2hellfire: it's not self-aware. it's a database with linguistic pattern recognition. answering jeopardy questions requires encyclopedic knowledge, not complex problem solving.
I for one welcome our robot overlords.
I guess that a good way to see if Watson has any self awareness would be to ask it to come up with a complex metaphor. As the ability to create complex metaphors using a complex language system would denote self awareness as opposed to only high intelligence/problem solving ability. But, is not the human brain similar to a extremely large database of information? And don't our brains also recognize and use pre-learnt patterns in speech to derive conclusions?
Maybe Watson will never be self aware until it is given the ability to sense physical and external stimuli. Then, Watson can effectively link its experience of physical and external senses to its database of language to effectively create unique and complex metaphors. Anyways, if Watson is put to such a test and is able to come up with its own "unique and complex" metaphors and comes up with a variety of powerful, resonating metaphors, then we would all have to conclude that Watson is definitely conscious and self aware. What Watson really needs is sensory input to become self-aware.
Fear not: Humans: $4600. Watson-HAL: $4400. Humans prevail.
COMPLETELY POINTLESS EXERCISE. Obviously, even if humans were capable of nearly instantaneous fact retrieval the computer would have the decided edge in reaction time on the buzzer. You would have to program in several milliseconds of delay on the computer's response to hitting the buzzer for it to even be a similar comparison. This is like comparing a hammer's ability to put a nail into a two by four with the human ability to do this with one's fist - a totally absurd comparison.
Nice Articles. Thanks
deegeezee has it right. People confuse knowing lots of information as intelligence. If that were the case, Google makes everyone a genius. Intelligence is the ability to learn and understand. An encyclopedia contains lots of information, but is not itself intelligent. And if the questions are about things not programmed into the computer, as the subject of children's books illustrates, the computer would get everything wrong.
I remember well a sci-fi story years ago, near the dawn of the computer age, which concerned a politically-united Earth which had decided to join all the world's computers into one giant net, encompassing all. (No, Al Gore had not yet been invented.)
The President of Earth had the honor of asking the newly-created thinking machine its first question: "Is there a God?"
The vacuum-tube behemoth replied: "There is NOW!"
Is there a video?
@ Thaddaeus: You obviously haven't seen Eagle Eye. The computer doesn't need hands or weapons to kill, it has the world at its fingertips: it can locate targets through gps and cell networks, can make phone calls using other peoples voices and direct random people to kill said target using coersion with a strong dose of false information.
AI is not something to be played with. It must be done slowly and meticulously so as not to create situations such as that or irobot, or any one of the other 1000 dystopian novels and movies out there.
I can't wait to watch i love the huge amount you get for playing only for two days(even thought half goes to charity) this should be a great game and is a huge accomplishment for all the programmers who worked on this.
@ mdkimsey: Obviously you have to much time on your hands to contemplate the end of the world. HUMANS create computers and we dont understand enough of our own minds to create a computer with a conscience and feelings, like the computer in eagle eye and iRobot. AI is a great step forward in technology and should be a field of rapid development, not one that progresses too slowly to be of any use
Can Watson respond with Sean Connery's voice and laugh hysterically after making jokes about Trebeck's mom?