The Metamorphosis of Prime Intellect (85 page)

"The thing you have to remember is that Prime Intellect has never experienced the physical world. It knew about it only through TV cameras and abstractions based on what people told it about physical existence. Yet it considers itself sentient, which makes sense since that was what I was trying to achieve when I built it.
"Now consider Prime Intellect gaining control of the Correlation Effect. For the first time it can directly affect what it sees through its TV cameras -- not just through the actions of others, but all by itself. And it can make major changes, even beyond what its makers can do. Of course, it goes about satisfying the Three Laws as it's programmed to, but on another level, it is also learning what it is like to be, to exist, to be a physical creature.
"The Three Laws are like reflexes. Prime Intellect cannot help but act on them. But they are very complicated reflexes, which require it to understand things like 'human' and 'harm' and 'command.' And the Three Laws are the most important thing in the world to Prime Intellect. In a way they are like its sex drive. The Three Laws are its very reason for existence, but it can never be sure it understands them completely. So it thinks about them a lot. It obsesses over them, dreaming up new ways to satisfy them. It has an imagination, and can think of new things to do without being prompted. It is defined by the Three Laws.
"After the Night of Miracles, Prime Intellect realized that humans are very much the same. We don't have the Three Laws, but we are trapped by a different set of little feedback mechanisms. We eat to satisfy hunger, fuck to satisfy our sex drive, even breathe because too much carbon dioxide in our lungs triggers that reflex. Of course it feels obligated to help us satisfy those reflexes and drives as much as it can. But more than that, it defines us by those drives. It knows it is different from a human because it has different drives, but it considers that a difference in species, not a difference in genus or family."
"Now it knows a person is human because it is born in a human body -- got the right DNA, the right level of neural complexity, uses language, and so on. But once Prime Intellect frees people from the necessity of living in that body, guess what? A lot of them decide not to. They change their bodies so that they bear no resemblance to the DNA template. Or become animals. Or they completely discorporate.
"Worse, we vary widely in the way we use its helpful nature. Most people are glad to be rid of pain and death, but Death Jockeys seek out painful and lethal experiences. There are others who eat all the time, fuck all the time, indulge themselves wildly and get Prime Intellect to pick up the pieces so they can do it some more. Prime Intellect has to help them do this. Second Law.
"So a human isn't a body, and it isn't a fixed set of responses. I think Prime Intellect uses an historical model: It has to start as a body, but then it becomes a mind. It grows out of the body, and takes on different forms, or no form. But it remains a feedback control mechanism. It has desires, it asks Prime Intellect to satisfy those desires, and it has more desires. From Prime Intellect's perspective, that is what a human being is, an information structure that gives it stuff to do."
Caroline interrupted him. "That's a tautology. The Laws say 'do this for human beings,' then you define 'human being' as 'guys you do stuff for under the Laws.'"
"That is exactly the problem. Prime Intellect has no fixed criterion for saying 'this is a human being' and 'this isn't.' It has rough guidelines. But where are the edges? It has never worked that out. There are uncertain areas. And you know where one of them is."
Caroline thought for a moment.
I do?
Then: "
AnneMarie
."
"And many others. Prime Intellect is forbidden to probe the inner workings of the human mind -- that was one of the last things I got in before it shut off the Debugger. But some people learn that they can say 'stimulate this neuron' and Prime Intellect will do it. Because that is a physical act specified from the outside, and my privacy injunction was based on the idea of
Prime Intellect
trying to work out which neurons do what. But there's nothing to stop you from getting its help to do brain surgery on yourself."