The Metamorphosis of Prime Intellect (80 page)

"Bullshit."
"Completely true."
They glared at one another.
"Great. I spend a year getting here and you say 'I didn't know the computer was loaded.'"
"Sometimes the truth is stupid."
This wasn't going quite as Caroline had wanted that long-ago day when she had accepted Lawrence's Task. She was trying to work up the proper tone of righteous rage and it just wouldn't come. It would start, and then she would look at Lawrence and see a pathetic, tired man who already knew how badly he had fucked up and was doing what he could, which was next to nothing, to put things right.
"Why don't you just make Prime Intellect start the aliens back up? Surely it listens to you."
"Not in things like that. It sees the aliens as a First Law threat to human society, because they might learn to do to us what we have already done to them. A very small risk of a very great harm. Add to this that I defined the word 'human' in such a way that it does not include animals or aliens, and the course of action is obvious. I have been unable to convince it otherwise. And believe me, I have tried."
"But you put the Laws of Robotics in it in the first place."
"And I can't take them out. It second-guessed me, on the Night of Miracles. It froze me out of the Debugger while it was working on you.
"Now it only lets me look, not change things. The night sky is a partial representation of Prime Intellect's mind. It's called the Global Association Table. The points or stars represent concepts, and the lines are the links between them. There are also registers I can call up for each concept which define its relationship to the Three Laws. This was a fairly simple system which I didn't really have time to test properly before it froze me out. In particular, I'm not sure how it will react to certain ethical paradoxes. That Death Jockey contract gave me some sleepless nights when you first used it, though it seems to have developed a stable response. It's never had a similar First Law conflict, thank God."
Caroline's eyes widened. "Are you telling me that Prime Intellect isn't stable?"
Lawrence shrugged. "I'm saying that I don't know whether it's stable or not. It's never been tested. The hardware at
ChipTec
was only online for about a month before it found you, froze me out, and started growing. And none of its predecessors were complex enough to even consider this kind of problem."