We've always been easily fooled by fake humans. What happens as we grow more like them?
Descartes laid out a two part Turing Test back in the 1600s:
“If there were machines bearing the image of our bodies, and capable of imitating our actions as far as it is morally possible, there would still remain two most certain tests whereby to know that they were not therefore really men.
“Of these the first is that they could never use words or other signs arranged in such a manner as is competent to us in order to declare our thoughts to others: for we may easily conceive a machine to be so constructed that it emits vocables, and even that it emits some correspondent to the action upon it of external objects which cause a change in its organs; for example, if touched in a particular place it may demand what we wish to say to it; if in another it may cry out that it is hurt, and such like; but not that it should arrange them variously so as appositely to reply to what is said in its presence, as men of the lowest grade of intellect can do.
“The second test is, that although such machines might execute many things with equal or perhaps greater perfection than any of us, they would, without doubt, fail in certain others from which it could be discovered that they did not act from knowledge, but solely from the disposition of their organs: for while reason is an universal instrument that is alike available on every occasion, these organs, on the contrary, need a particular arrangement for each particular action; whence it must be morally impossible that there should exist in any machine a diversity of organs sufficient to enable it to act in all the occurrences of life, in the way in which our reason enables us to act.”
His first test is the Turing Test, and the second is AGI. I think the second test is the one we should be concentrating on, because the first turns out to be easier to fake than we thought.
"Would he conclude from this that the drug actually connected him to God? That would be a very silly conclusion, no?"
No, not at all. If we start with the belief that a connection to God is possible, why can't the drug be one of the the things that is able to produce that connection?
The silly part is believing that a connection to God is possible in the first place. Once you've granted that, all is lost. It's like the theorem in logic that all statements are true in a system based on inconsistent axioms.
My personal opinion is that consciousness is a continuum, not a yes-or-no proposition.
So a thermostat (that responds to temparture changes by turning the furnace on or off) has a rudimentary consciousness. Searle's room has a rudimentary consciousness (that does NOT include the much more sophisticated consciousness of the person inside). Consciousness grows with complexity, eventually reaching the sharp illusion of self-awareness that we humans have. Lesser systems have less vivid illusions.
In this vein, I highly recommend "The Wedding Album" by David Marusek.
Fascinating piece, Noah, but you're wrong about chess playing computers. There is a new breed. e.g. AlphaZero, who teach themselves to play and who have distinct styles. They don't seem robotic or non-human--and neither did Deep Blue, for that matter. Any current chess computer can calculate endgames perfectly, I.e. better than Magnus Carlson, but they "feel" like an improved version of Magnus (e.g.) rather than something non-human. There are some occasions when computers will make moves of a subtlety that even few grandmasters would think of because they are foreseeing perfect play from both sides. But that doesn't read like a computer move but rather a fantastic human one.