Obscure intentions of a dangerous mind

But it is not computers' commercial nous or Siri that Hawking and Musk are concerned about; they are wary of the approaching so-called singularity, the point when artificial intelligence will exceed human intellectual capacity and control. To be truly useful, artificial intelligence must be able to make decisions, but such autonomy also means the ability to act independently.

It seems reasonable to suppose this will mean artificial intelligence which is self-aware, and there will be a dramatic difference between very smart software and intelligent software that thinks, therefore it is. And at that point a very stark difference between it and us - one that will only grow wider.

As Hawking said, "It would take off on its own, and re-design itself at an ever increasing rate ... Humans, who are limited by slow biological evolution, couldn't compete, and would be superseded."

Even if we are just trying to simulate human intelligence, the outcome of a self-aware machine is likely to be very different from what was intended given that we know so little about what goes on inside our heads.

And at some point how will the designers know if what they are designing is not being programmed by themselves. They could be within a feedback loop that is designed to evolve itself.

A self-aware artificial intelligence is unlikely to show much empathy with us, it will be something totally other; at best it will see us as some kind of pet, perhaps some kind of pest, or more likely it will know we are the biggest threat to its existence and seek to protect itself.

At the moment we are at the embryonic stage of artificial intelligence, and although people like to use the quip - why worry, we can just pull the plug - it is not likely to make itself known to us by coming bawling out of its cyber womb or by saying "mama"!

The author is a writer with China Daily. hannayrichards@chinadaily.com.cn