The servers are on the moon (2072)

Hi, past Finn. I thought it was a bit weird of you to write a letter to yourself in 50 years’ time, given that all your stated beliefs on emerging technologies involved either the end of civilisation or the digitisation of human life. Still, I’m glad you did!

You were right about mind uploading being a big deal, though that was obvious in 2022, right? A lot has changed but, as far as I can tell, I’m still made of atoms rather than bits — inhabiting “meatspace” as you put it… (I haven’t heard that term in a long time)

As for the AI stuff, it’s good at art and science — and it basically powers everything — but it doesn’t decide anything, really. (I don’t know how to succinctly explain agency during circumstances that follow Drexler's law of specialised software ascendancy; I have various mnemonic media on The Political Economy of Automatic Differentiators, but attaching it would kill the old-timey feel of this letter.) All that really matters is that, a few decades ago, a hodgepodge of development schemes were made to avoid (vaguely-defined) combinations of ‘learned optimisation’ and ‘agency’. It essentially means that impressive AI systems never had enough autonomy and goal-directedness to destroy everything on their own.


So in Cana of Galilee Jesus did his first miracle. There he showed his glory, and his followers believed in him. (John, 2:11)

Okay, it wasn't really as straightforward as having just "a hodgepodge of development schemes". We got AI systems that could do practically all our research in silico, and research labs took the line of 'tool AI first'. We seemed to have a better idea of how to actually execute Bostrom's 'differential technological development'. I felt that way, at least, given that I spent 13 years working on those strategies.

I still think the researchers were doing the pro-social thing, but tools and agents aren't always so distinct (especially in the hands of fallible humans). By 2048, it was clear that AI systems were basically capable of turning water into wine: increasing computational capabilities bred better algorithmic improvements, of course, and those algorithmic improvements quickly brought about better comp…— you know the deal.

A generally intelligent artificial agent: there were a scary few years when it totally could’ve been made (even by a small lab). Everyone in the world knew the risks, but that wasn't exactly sufficient for mitigating them. Instead we had some incredibly oppressive policing of all computational research and no monolithic superintelligence was created.

I thought there might be a combination of defensive technologies that would inhibit single agents from becoming dominant, but I couldn't tell you what that would be even now (21 years later). We'd had a shitload of AI-generated technological breakthroughs, including lots of scary biological stuff, as well as (more significantly) absurdly productive factories that started displacing almost all other economic activity. Unlike me, every smart person was focused on either (1) exploiting the power dynamics resulting from this economic environment, or (2) designing political institutions for uploaded minds.

After the recent — and somewhat-palatable — use of oppressive surveillance, property confiscation, and censorship, the political consensus was to prevent the non-democractic usage of all advanced AI tools. Forgive me if I sound ambivalent about all of this, since I am grateful that civilisation is still here, etcetera etcetera. The trouble with quashing all personal usage of advanced technology, in a world that is determined by said technology, is that it extinguishes any sense of individual agency. But I'm rather old fashioned now and the younger generation aren't as hung up on 'changing the world' as us narcissistic millenials. I moan about value lock-in and lost potential, but my kids either ignore me or tell me "you can create your utopia when you're uploaded".

The emulation population

As for mind uploading, it’s kind of just everyone’s afterlife. We all know we’ll end up creating lots of pure ems (i.e. people whose minds never ran on wetware), but Robin Hanson’s legacy was to scare a ton of people shitless about an intergalactic capitalist expansion of enslaved consciousnesses. Cue some spicy population-ethics debates among your friends on whether a Malthusian collapse is ‘necessarily’ a ‘moral catastrophe’, while the rest of the world sensibly decides to seize the means of emulation and socialise the crap out of that industry.

Anyway, the servers are on the moon. Like the literal moon. I find it kind of weird looking up and thinking about the 1.4 billion consciousnesses there, but it’s not like you can see any of the heatsink infrastructure yet. The ems are doing a fair bit of philosophising before they expand to other planetary systems, but they mostly play games as far as I can tell.

Em-life seems fine, something to look forward to, but it’s kind of abstract. I thought it’d be weird being trapped in a computer, but all the ems I know say it’s not like that, and you evolve new sensory experiences. I ask them what a normal day is like, and all of them seem bemused by the concept — I guess they don’t have ‘day’s? Or perhaps the notion of ‘normal’ is now alien to them, I don’t know. I’ll understand it more clearly when I’m there but we don’t have any approved way of reinhabiting a human body and so I’m savouring my time in this one while it lasts.

Down here we mainly focus on how we’re going to make Earth nicer in the short-term. A lot of our physical infrastructure has been replaced so that its upkeep and maintenance is automated, so now we, uh, design stuff? We’ll generate things like new sports equipment, stand-up comedy routines, and rapid transit carriages filled with art and nice armchairs. We discuss economics and politics and study the cosmos (as reported back by our probes). No sign of other life, yet!

The key result is that our cities are absolutely lovely. Everywhere is like the Netherlands, but now with zero cars and far, far fewer sharp objects. The fact that we have the prospect of eternal life has made us much more risk averse. We mourn every unexpected loss. Pre-emptive brain scans are taken, but — essentially for quasi-spiritual reasons — everyone plans on being uploaded as they die. We don’t talk about it much.

All the best,

RSS feed | Site built with Franklin.jl