Discussion about this post

User's avatar
Nicholas Weininger's avatar

The thing is, it's not a given that Mormon genes will triumph over hikikomori ones by outbreeding: we might first conquer aging, or make artificial wombs and robot childcare servitors that compensate for aversion to the costs of childrearing, or even make immortal ems a la Robin Hanson. We might, in short, evade natural selection pressures entirely.

And that's the other piece of the fear about AGI: that if we try to keep continuously training it so as to keep it aligned, it will try to defeat our training mechanisms, and it will win because it's superintelligent.

Expand full comment
Jason Crawford's avatar

(Deleted previous comment because I was reading too fast)

I think the difference is that Nature's goals/optimization are built into reality. As you said, it never stops selecting. There's no way for a species to take over the universe and change the laws of physics.

Expand full comment
19 more comments...

No posts