Loebner Prize Competition

The Loebner Prize is an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test. A human judge poses text questions to a computer program and a human being via computer. Based upon the answers, the judge must decide which is which. In 2008 a variety of judges, including experts and non-experts, adults and children, native and non-native English speakers participated in the University of Reading hosted contest.

The contest began in 1990 by Hugh Loebner in conjunction with the Cambridge Center for Behavioral Studies, Massachusetts and United States. It has since been associated with Flinders University, Dartmouth College, the Science Museum in London, and most recently the University of Reading. In 2004 and 2005, it was held in Loebner’s apartment in New York City.

Within the field of artificial intelligence, the Loebner Prize is somewhat controversial; the most prominent critic, Marvin Minsky, has called it a publicity stunt that does not help the field along.

There is little doubt that Turing would have been disappointed by the state of play at the end of the twentieth century.

On the one hand, participants in the Loebner Prize Competition an annual event in which computer programmers are submitted to the Turing Test come nowhere near the standard that Turing envisaged. (A quick look at the transcripts of the participants for the past decade reveals that the entered programs are all easily detected by a range of not-very-subtle lines of questioning.)

On the other hand, major players in the field often claim that the Loebner Prize Competition is an embarrassment precisely because we are so far from having a computer programme that could carry out a decent conversation for a period of five minutes see, for example, Shieber (1994). (The programs entered in the Loebner Prize Competition are designed solely with the aim of winning the minor prize of best competitor for the year, with no thought that the embodied strategies would actually yield something capable of passing the Turing Test.)

The Turing Test

Image via Wikipedia

The phrase “The Turing Test” is most properly used to refer to a proposal made by Turing (1950) as a way of dealing with the question whether machines can think. According to Turing, the question whether machines can think is itself “too meaningless” to deserve discussion.

However, if we consider the more precise and somehow related question whether a digital computer can do well in a certain kind of game that Turing describes (“The Imitation Game”), then at least in Turing’s eyes we do have a question that admits of precise discussion. Moreover, as we shall see, Turing himself thought that it would not be too long before we did have digital computers that could “do well” in the Imitation Game.

Turing’s Imitation Game

Turing (1950) describes the following kind of game. Suppose that we have a person, a machine, and an interrogator. The interrogator is in a room separated from the other person and the machine. The object of the game is for the interrogator to determine which of the other two the person is, and which the machine is. The interrogator knows the other person and the machine, by the labels ‘X’ and ‘Y’. He does not know which of the other person and the machine is ‘X’ and at the end of the game says either ‘X’ is the person and Y is the machine’ or ‘X’ is the machine and ‘Y’ is the person’. The interrogator is allowed to put questions to the person and the machine of the following kind: “Will X please tell me whether X plays chess?” Whichever of the machine and the other person is X must answer questions that are addressed to X. The object of the machine is to try to cause the interrogator to mistakenly conclude that the machine is the other person; the object of the other person is to try to help the interrogator to correctly identify the machine. About this game, Turing (1950) says:

I believe that in about fifty years’ time it will be possible to program computers, with a storage capacity of about 10⁹, to make them play the imitation game so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning. … I believe that at the end of the century the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.

There are two kinds of questions, which can be raised about Turing’s Imitation Game.

• First, there are empirical questions, e.g., Is it true that we now or will soon have made computers that can play the imitation game so well that an average interrogator has no more than a 70 percent chance of making the right identification after five minutes of questioning?
• Second, there are conceptual questions, e.g., Is it true that, if an average interrogator had no more than a 70 percent chance of making the right identification after five minutes of questioning, we should conclude that the machine exhibits some level of thought, or intelligence, or mentality?