It’s fair to say that few people are currently thinking about the future of Artificial General Intelligence (AGI). Those who do are mostly researchers and engineers working on the topic.
Wikipedia article – Existential risk from artificial general intelligence
The hope is based around alignment, a term used broadly to mean the degree to which an AGI can be made to conform to human goals and objectives. Poorly aligned AGI would probably be diasastrous and unsafe, well aligned AGI might be beneficial and safe. And let’s be clear at the outset: there are concerns about how AGI (and even current technology like ChatGPT) will cause disruption and harm by affecting social behaviour or employment in the way new technologies have done in the past. But that’s not what we are addressing here. The more significant thinking is about the existential threat to humanity. Will an AGI spell the end, will it render humans extinct?
Nor are we discussing here whether or not it will be possible to develop an AGI, some people argue not, that there’s some kind of unbridgeable leap between current AI systems and an intelligent system with thoughts, ideas of its own, and self-awareness. But we know this is incorrect; such a system has already been created by evolution – humans! This level of intelligence is achievable and it’s already been done. It may take time, it might not be easy, but developments in neural networks and related systems will lead to AGI sooner or later. ChatGPT can already write workable computer code – just think about that for a moment.
The hopeful news
The hopeful news comes from OpenAI, the company that created ChatGPT and the GPT software behind it. They have found that process supervision produces better results than outcome supervision. And this gives us a much better chance of understanding how the AI makes its choices. Process supervision feeds back on the quality of the interim stages of an AI’s processing; but outcome supervision feeds back on only the quality of the final result.
Understanding how an AGI works might help developers build in robust alignment features; and if process supervision is more effective than alternatives, it stands a good chance of being employed by developers.
The less hopeful news
The unhopeful news comes from discussions about the nature of risk and human thinking about risk, expressed in discussions on LESSWRONG and elsewhere.
Closing thoughts
Currently, only a very small number of people are concerned about a possible existential threat from AGI. But most of the people with that concern are some of the same people that have knowledge and experience of AGI, what AI can currently do, and how quickly the systems might advance and escape our control.
Personally, I think the threat is real and potentially impossible to stop with very little time for effective countermeasures. But I also think it might just be possible to avoid the danger, but only if we have binding international agreements and strong oversight. However, I further suspect that it might be impossible to agree those necessary measures completely enough, rigorously enough, or fast enough to be fully effective.
In my judgement we are on rather shaky ground, and the more people who give this serious thought as soon as possible, the better. When the stakes are so very high there can be no such thing as too careful or too thorough.
See also:
- JHM – The end of humanity?
- OpenAI – Improving mathematical reasoning with process supervision
- LESSWRONG – Uncertainty about the future does not imply that AGI will go well
- Wikipedia article – Existential risk from artificial general intelligence
Journeys of Heart and Mind 