Now, the next step is where I personally have my doubts about the Singularity.
By my calculation, in 2010 we will be able to build a computer as intelligent as a human, using only established technology (45nm chips aren't in mass production yet, but they have been made) and existing knowledge (it's a cell-function simulation of the brain; we don't need to understand anything about, say, consciousness; all we need to know is what the cells do and how they are hooked together).
That's a fairly large leap. I don't think it'd be like hooking billions of muscle cells together and getting a contraction. Consciousness may be more like ten million Mona Lisa paintings, all different, all coming up and functioning together at the same time.
Aside: Obviously it will be enormously difficult to build and program this machine, but we can start work now, because nothing much is going to change.
Yes, that is the problem.
The next step is this: In 2010, we can build a computer as smart as a human for $200 million. So by 2012 or so, when the 35nm process is in full production, we will be able to do it for $100 million... Or we can build a computer twice as smart for the same amount of money.
Strong vs. weak AI, which is to say, how inherently smart is the machine, and how fast can it think? An average human who thought a million times as fast as me might be able to beat the crap out of me easily, but could not beat me in an IQ test.
One of the key ideas of the Singularity is that once you have a computer twice as smart as a human, things begin to take off by themselves. The computer itself takes a hand in the research program. If it can come up with a modified neuron-simulation algorithm which is 50% more efficient than what we normal humans managed, then suddenly it is three times smarter than us. And then the new 3x computer can come up with better ways to build the chips or whatever...
Personally, I strongly suspect that we will run into the law of diminishing returns here. I think that a computer "twice as smart" as a normal human will be less than twice as capable at solving problems, and also that the problems associated with developing ever smarter computers will become less and less tractable. I could be wrong, of course.
Given many types of problems cannot be solved via shortcut, i.e. they must be solved by essentially simulating things then running time forward, I think such hyper-intelligences would have massive auxiliary experimentation virtual realities.
Trivially, and I've read a sci-fi story recently about this, an evil scientist captures normal human minds downloaded into a computer, resets them, and runs them as they do research, thinking they're normal humans in a normal world but highly fast forwarded, and he has hooks into all their virtual computers and simply reads the papers they type, then presents them as his own work in the real world.