Life as we don't know it

The report looks past the milestone of human-level AGI and asks how AI itself might keep developing in a post-AGI world. The endpoint of that continuum — Universal AI — is theoretically well understood, which lends some formal grounding to an otherwise speculative question.

Determining whether these frictions will be negligible or substantial raises a number of concrete open research questions.

AGI, ASI, UAI

AGI — human-level

Shorthand for human-level artificial general intelligence: a system roughly as intelligent as a single human, at median human level on most cognitive tasks. The first AGI will already be superhuman on many narrow tasks, but not yet general enough.

ASI — superhuman & general

Artificial superintelligence has superhuman ability across virtually all tasks and domains. The report sets the bar high: a system that exceeds large, well-coordinated human-expert collectives. Narrowly superhuman systems like AlphaFold or AlphaGo are therefore ruled out as ASI.

UAI — the theoretical limit

Universal AI is the theoretical endpoint, formalised via the AIXI agent, which by definition maximises the Legg-Hutter score of intelligence. It is incomputable, and can only be approximated from below by ever more powerful ASIs.

A universal yardstick

The Legg-Hutter score formalises intelligence as the average performance of an agent across all computable tasks, with simpler tasks (lower Kolmogorov complexity) given more weight.

Crucially, this makes intelligence a continuum. We do not need a sharp threshold separating AGI from ASI — what matters is that there is a large gap in score between them, within which the pathways and frictions of this report play out.

Caveats worth keeping

  • AIXI is a learning algorithm — the right comparison is against an architecture and training algorithm, not a single trained model.
  • The task set could be narrowed to tasks of 'current and future human interest' rather than all computable tasks.
  • Real systems may be 'jagged': superhuman on some tasks while sub-human on others.
  • These definitions are relative to a human baseline that itself moves as tools and knowledge improve.

Hard limits — even for ASI

Superintelligence is not omnipotence. Even far beyond human-level AGI, some ceilings cannot be engineered away (Table 2):

  • Fundamental physics — e.g. the speed of light.
  • Real-time constraints on acting in the world.
  • Limits on physical manipulation of matter.
  • Irreducible epistemic uncertainty.
  • Computational complexity — e.g. P vs NP.
  • Logical limits — e.g. Gödel's incompleteness.

So an ASI will not necessarily cure ageing, reshape matter with nanobots, upload human brains, or build Dyson spheres. Predicting where progress plateaus, and at what capability level, remains genuinely hard.