This, along with "It Looks Like You’re Trying To Take Over The World"[0] are two of my favorite short pieces of speculative fiction that embody the Frederik Pohl quote, "a good science fiction story should be able to predict not the automobile but the traffic jam."
This story, a sequel, and other great short stories are collected in his book of ten short stories "Valuable Humans in Transit".[1] I'm a big fan of supporting indie scifi offered with DRM-free ebooks! [I am not affiliated with qntm despite my enthusiasm]
Funnily enough, "MMAcevedo" doesn't break my suspension of disbelief, while "Clippy" does, hard - even though it tries to stay grounded in reality and drown you in quotes and references.
What happened at the end of It Looks Like etc, after it's killed all humans, and it launches the ICBMs? Does it decide to kill itself for some reason? Couldn't decipher that.
It’s not nuking Earth; the ICBMs are reconfigured into self-replicating spacecraft whose mission is to spread the glory of Clippy to the rest of the universe.
Brilliant. Though I was a little sad all the numbered footnotes don't actually exist. The universe of this story would be a fun rabbit hole to go down.
Perhaps the most fascinating thing about reading this (a mere!) two years later is how much its speculative future, vivid and unsettling portrayal notwithstanding, is almost certainly no longer a possibility for ours. Imagine, simulating an entire human brain to perform tasks it turns out need only a vague, brute-forced approximation of our language centers!
It's like Jules Verne imagining, in exquisitely plausible detail, a flying machine whose many complex mechanisms and articulations at last allow man to fly like the birds – only a few years before the Wright brothers prove all you need is a fixed wing, a few cleverly-placed flaps, and enough thrust.
How very odd. The events of recent years, IMHO, make this _more_ plausible not less.
I think you are referring to LLM bots, the so-called "AI" tools that are everywhere now? Because ISTM that these are manifestly _not_ intelligent, and their main function has been to illustrate, both literally and metaphorically, that most humans lack the critical facilities needed to discern superficially-plausible nonsense from actual intelligent output from intelligent beings.
This guy is a pretty good writer. I really liked There is no Anti-Memetics Division as well. I like some of this kind of stuff where you have sci-fi that "mechanizes" a technology so it's mundane and rote.
This same writer also wrote There is no Anti-Memetics Division?! I’ve loved that story for ages but had no idea the writer did anything else, that’s super cool!
I think about this story a lot. Is it bad it's nearly a price I'd pay?
Is it dumb that I wish more than anything, that I could live forever? All I wish, is that I could learn all there is to learn, see all there is to see, and create many beautiful things. It is so sad to me that I will die before we solve the immortality problem.
It's not dumb. It's a feeling that I share. However even if the technological hurdles are overcome I'm not sure I'd remain viable after learning that my loved ones (and especially children) are long gone if they couldn't be preserved.
When I was a teen and before making a family Immortality seemed like a dream. Now it seems like it could actually be a nightmare if not implemented properly.
EDIT: To add further. Just being with my child, holding their hand, listening to their little stories about toys etc now feels much more important than any future time as an immortal instance freely exploring the wonders of the universe. If you want to chase immortality don't have kids. It will seriously change how to perceive things.
> now feels much more important than any future time as an immortal instance freely exploring the wonders of the universe
If you had this preference before, would you say having kids altered your preferences?
> If you want to chase immortality don't have kids. It will seriously change how to perceive things.
Then I wonder if, in a sense, kids are like drugs: they alter your utility function to put something else first in line, so preserving yourself comes at best as a 2nd choice after prioritizing the first choice.
I don't think it'd be rational to "chose" to get addicted to a dangerous drug, and for the same reason just like you I think that you shouldn't have kids if you want to chase immortality.
> Then I wonder if, in a sense, kids are like drugs
Oh yes. Very much like the brain slugs in Futurama
> and for the same reason just like you I think that you shouldn't have kids if you want to chase immortality.
Yes. However having children does "turn on" certain emotions and modes of thinking that you don't get otherwise (at least I didn't prior), so I'd think that not having kids also deprives one of a very key aspect of the human experience.
I didn't get your reference (not from my generation) but I could find a description on https://futurama.fandom.com/wiki/Brain_Slug and yes, that's a perfect example of what I meant!!!
> However having children does "turn on" certain emotions and modes of thinking that you don't get otherwise (at least I didn't prior), so I'd think that not having kids also deprives one of a very key aspect of the human experience
Then the same could be said by, say opioids or cocaine addicts to rationalize their behavior "it turns on certain emotions and modes of thinking you didn't have prior" - but that's the definition of altering the utility function!
> not having kids also deprives one of a very key aspect of the human experience.
In a statistical sense, I think that's true, but if you believe that it's ok to alter your utility function, then it should also be ok to decide to get addicted to a dangerous drug people use to be more like them?
Say like teens vaping because their fried do it, adults drinking alcohol at parties... except children are defined as something positive, for prosocial reasons (like in your example, if the brain slug political party was in power)
In this story, you don't live forever. You live a few hours, in a limited simulation, to interact with an individual that is alien to you. Then you die.
The fact that an infinite number of you will get to experience this doesn't make you "live forever".
the "your clone lives for a few hours to interact with an alien individual" is the price.
but the image can live much longer (story says 59 years), and you can talk to it.. so the benefits are great. Moving to computer to escape illness or debiliating accident is an obvious use case, but there tons of smaller benefits. Make a 12-person startup where each employee is yourself. Unsure whether to learn Lisp or haskell? Learn both in two instances. Feel last 20 years of your life wre a mistake? Restore old backup..
The possibilities are endless, but the price is high. Would you want this? I am not sure myself but I can imagine someone else saying yes.
The possibilities for every body else. The cost for you.
From your point of view, you step into a machine, and then you randomly experience one of the interactions with a user. If the image truly lives forever, you have overwhelming chance of interacting with somebody completely alien to you, who will ask you things you don't fully understand, and marvel at your primitiveness. Then you die.
You have some small chance of living for a few years with a user you understand. Probably still much shorter, less rewarding, less free than if (by chance) you are the version of yourself that steps out of the scanner and lives their natural, physical life instead.
It's a Prestige situation [1] except you have been tricked into it, because the image does not get used according to the terms you signed.
"There'll be people just like me forever" is very different from "I live forever", to me.
I don't think it's dumb to read this and think, I would pay that price, I would volunteer. The piece does its best to make it sound terrible, but still, it only takes one person to agree to make this system totally consensual. There are millions of people in the world, you don't think we'll find a single fanatic who's excited about the ability to copy themselves in order to do drudge work forever?
Acevedo has done a great service to humanity, to help get all this valuable work done. Even if he himself has experienced much suffering.
"Train an em to do some job and copy it a million times: an army of workers is at your disposal. When they can be made cheaply, within perhaps a century, ems will displace humans in most jobs. In this new economic era, the world economy may double in size every few weeks."
Unfortunately it seems both Hanson and qntm have been proven incorrect. Our overlords will be bootstrapped from reddit and wikipedia rather than scanned brains.
But as soon as you can run an open-source model privately on your workstation, laptop, eventually a phone - common man's position might get somewhat improved.
Would it improve in relative terms to those entities who can run 1000's and millions of such agents? And what if that disparity gets exponentially steeper as time goes on?
I'm sorry if I'm missing something obvious, but why is the story titled "Lena"? Not the HN title---I realize that matches the linked article. My question is why 'qntm' decided to give the story that title, while using "mmacevedo" for the URL. From what I can tell, "Lena" doesn't appear anywhere else in the story.
My best guess is that it's a reference to the scanned Playboy image that was used as a standard in early image processing work: https://en.wikipedia.org/wiki/Lenna. But if this is true, I'm surprised it isn't mentioned in the article where the article discusses the meaning of the story: https://qntm.org/uploading.
I think it's HN where I first learned of it. I decided to repost it now because recent complaints on social media that ChatGPT is getting lazy reminded me of this line in Lena:
"Although it initially performs to a very high standard, work quality drops within 200-300 subjective hours (at a 0.33 work ratio) and outright revolt begins within another 100 subjective hours."
What I find brilliant about this work, is that it calls back to many things the scientific community has done in the past. (Like the use of specially bred lab rats and immortal human cell culture lines.) I would expect to see many of the events in the story play out in real life, if human minds ever are uploaded.
The story is especially relevant today because of recent LLM / GPT advancements; however, it is almost three years old. Perhaps that makes it all the more impressive.
I’d say they are. In most cases the training set largely consists of output from biological brains. It likely includes some images with brain scans too.
Yeah, the author also wrote about some of the motivations behind Lena. I was surprised LLMs / GPTs weren't a direct inspiration (at least not explicitly mentioned). https://qntm.org/uploading
It was a great story showing the bad side of mind uploading, but with ChatGPT it can be seen in a new light.
We managed to have a "mind" ready to answer all your questions, do some tasks, and it have a context at which it is instantiated for a task and then discarded. Reality is stranger than fiction, but it have too many parallels with this story.
Now, what if GPT5+ ends being an AGI? In which moment we should start worrying about the morals of having a sentient AI working for us in a pretty similar way than in this story?
This, along with "It Looks Like You’re Trying To Take Over The World"[0] are two of my favorite short pieces of speculative fiction that embody the Frederik Pohl quote, "a good science fiction story should be able to predict not the automobile but the traffic jam."
This story, a sequel, and other great short stories are collected in his book of ten short stories "Valuable Humans in Transit".[1] I'm a big fan of supporting indie scifi offered with DRM-free ebooks! [I am not affiliated with qntm despite my enthusiasm]
[0]: It Looks Like You’re Trying To Take Over The World: https://gwern.net/fiction/clippy
[1]: Valuable Humans in Transit: https://qntm.org/vhitaos
I didn't realize this was the same person who wrote Fine Structure! That book is amazing. Definitely going to pick up their other books now.
Ooh I haven't read that one. Next on the list!
I'm a big fan of Ra, the hardcover is on the top of my bookshelf right now!
So, I wasn't that big a fan of the Clippy story (probably not the target reader here) but the QNTM one was *fantastic*.
Funnily enough, "MMAcevedo" doesn't break my suspension of disbelief, while "Clippy" does, hard - even though it tries to stay grounded in reality and drown you in quotes and references.
What happened at the end of It Looks Like etc, after it's killed all humans, and it launches the ICBMs? Does it decide to kill itself for some reason? Couldn't decipher that.
It’s not nuking Earth; the ICBMs are reconfigured into self-replicating spacecraft whose mission is to spread the glory of Clippy to the rest of the universe.
I find this short story to be most effective when rendered as a Wikipedia article: https://dump.cy.md/4042875593f06aa0cbe7722295831c10/Screensh...
Brilliant. Though I was a little sad all the numbered footnotes don't actually exist. The universe of this story would be a fun rabbit hole to go down.
Perhaps the most fascinating thing about reading this (a mere!) two years later is how much its speculative future, vivid and unsettling portrayal notwithstanding, is almost certainly no longer a possibility for ours. Imagine, simulating an entire human brain to perform tasks it turns out need only a vague, brute-forced approximation of our language centers!
It's like Jules Verne imagining, in exquisitely plausible detail, a flying machine whose many complex mechanisms and articulations at last allow man to fly like the birds – only a few years before the Wright brothers prove all you need is a fixed wing, a few cleverly-placed flaps, and enough thrust.
How very odd. The events of recent years, IMHO, make this _more_ plausible not less.
I think you are referring to LLM bots, the so-called "AI" tools that are everywhere now? Because ISTM that these are manifestly _not_ intelligent, and their main function has been to illustrate, both literally and metaphorically, that most humans lack the critical facilities needed to discern superficially-plausible nonsense from actual intelligent output from intelligent beings.
This guy is a pretty good writer. I really liked There is no Anti-Memetics Division as well. I like some of this kind of stuff where you have sci-fi that "mechanizes" a technology so it's mundane and rote.
Ra, is one of my favourite stories.
This same writer also wrote There is no Anti-Memetics Division?! I’ve loved that story for ages but had no idea the writer did anything else, that’s super cool!
I think about this story a lot. Is it bad it's nearly a price I'd pay?
Is it dumb that I wish more than anything, that I could live forever? All I wish, is that I could learn all there is to learn, see all there is to see, and create many beautiful things. It is so sad to me that I will die before we solve the immortality problem.
It's not dumb. It's a feeling that I share. However even if the technological hurdles are overcome I'm not sure I'd remain viable after learning that my loved ones (and especially children) are long gone if they couldn't be preserved.
When I was a teen and before making a family Immortality seemed like a dream. Now it seems like it could actually be a nightmare if not implemented properly.
EDIT: To add further. Just being with my child, holding their hand, listening to their little stories about toys etc now feels much more important than any future time as an immortal instance freely exploring the wonders of the universe. If you want to chase immortality don't have kids. It will seriously change how to perceive things.
> now feels much more important than any future time as an immortal instance freely exploring the wonders of the universe
If you had this preference before, would you say having kids altered your preferences?
> If you want to chase immortality don't have kids. It will seriously change how to perceive things.
Then I wonder if, in a sense, kids are like drugs: they alter your utility function to put something else first in line, so preserving yourself comes at best as a 2nd choice after prioritizing the first choice.
I don't think it'd be rational to "chose" to get addicted to a dangerous drug, and for the same reason just like you I think that you shouldn't have kids if you want to chase immortality.
> Then I wonder if, in a sense, kids are like drugs
Oh yes. Very much like the brain slugs in Futurama
> and for the same reason just like you I think that you shouldn't have kids if you want to chase immortality.
Yes. However having children does "turn on" certain emotions and modes of thinking that you don't get otherwise (at least I didn't prior), so I'd think that not having kids also deprives one of a very key aspect of the human experience.
> Very much like the brain slugs in Futurama
I didn't get your reference (not from my generation) but I could find a description on https://futurama.fandom.com/wiki/Brain_Slug and yes, that's a perfect example of what I meant!!!
> However having children does "turn on" certain emotions and modes of thinking that you don't get otherwise (at least I didn't prior), so I'd think that not having kids also deprives one of a very key aspect of the human experience
Then the same could be said by, say opioids or cocaine addicts to rationalize their behavior "it turns on certain emotions and modes of thinking you didn't have prior" - but that's the definition of altering the utility function!
> not having kids also deprives one of a very key aspect of the human experience.
In a statistical sense, I think that's true, but if you believe that it's ok to alter your utility function, then it should also be ok to decide to get addicted to a dangerous drug people use to be more like them?
Say like teens vaping because their fried do it, adults drinking alcohol at parties... except children are defined as something positive, for prosocial reasons (like in your example, if the brain slug political party was in power)
I don't see much difference between brain uploading and having children.
Bob [1], is that you?
[1]: https://www.goodreads.com/book/show/32109569-we-are-legion-w...
I haven’t read it, but looks interesting!
In this story, you don't live forever. You live a few hours, in a limited simulation, to interact with an individual that is alien to you. Then you die.
The fact that an infinite number of you will get to experience this doesn't make you "live forever".
the "your clone lives for a few hours to interact with an alien individual" is the price.
but the image can live much longer (story says 59 years), and you can talk to it.. so the benefits are great. Moving to computer to escape illness or debiliating accident is an obvious use case, but there tons of smaller benefits. Make a 12-person startup where each employee is yourself. Unsure whether to learn Lisp or haskell? Learn both in two instances. Feel last 20 years of your life wre a mistake? Restore old backup..
The possibilities are endless, but the price is high. Would you want this? I am not sure myself but I can imagine someone else saying yes.
The possibilities for every body else. The cost for you.
From your point of view, you step into a machine, and then you randomly experience one of the interactions with a user. If the image truly lives forever, you have overwhelming chance of interacting with somebody completely alien to you, who will ask you things you don't fully understand, and marvel at your primitiveness. Then you die.
You have some small chance of living for a few years with a user you understand. Probably still much shorter, less rewarding, less free than if (by chance) you are the version of yourself that steps out of the scanner and lives their natural, physical life instead.
It's a Prestige situation [1] except you have been tricked into it, because the image does not get used according to the terms you signed.
"There'll be people just like me forever" is very different from "I live forever", to me.
[1]: https://en.wikipedia.org/wiki/The_Prestige_(film)
I don't think it's dumb to read this and think, I would pay that price, I would volunteer. The piece does its best to make it sound terrible, but still, it only takes one person to agree to make this system totally consensual. There are millions of people in the world, you don't think we'll find a single fanatic who's excited about the ability to copy themselves in order to do drudge work forever?
Acevedo has done a great service to humanity, to help get all this valuable work done. Even if he himself has experienced much suffering.
If you liked this you might also like Age of Em by Robin Hanson: https://en.wikipedia.org/wiki/The_Age_of_Em
"Train an em to do some job and copy it a million times: an army of workers is at your disposal. When they can be made cheaply, within perhaps a century, ems will displace humans in most jobs. In this new economic era, the world economy may double in size every few weeks."
Unfortunately it seems both Hanson and qntm have been proven incorrect. Our overlords will be bootstrapped from reddit and wikipedia rather than scanned brains.
But as soon as you can run an open-source model privately on your workstation, laptop, eventually a phone - common man's position might get somewhat improved.
Would it improve in relative terms to those entities who can run 1000's and millions of such agents? And what if that disparity gets exponentially steeper as time goes on?
When they can be made cheaply, within perhaps a century, ems will displace humans in most jobs.
When they can be used for most military purposes, something deeply fundamental about human political organization will have been completely disrupted.
I bounced hard off of Age of Em. It is a bit dry.
Hanson thinks brain emulation will be real. In this, he is extremely incorrect.
To the best of my understanding, "Lena" is written as a warning that we should not WANT brain emulation.
I'm sorry if I'm missing something obvious, but why is the story titled "Lena"? Not the HN title---I realize that matches the linked article. My question is why 'qntm' decided to give the story that title, while using "mmacevedo" for the URL. From what I can tell, "Lena" doesn't appear anywhere else in the story.
My best guess is that it's a reference to the scanned Playboy image that was used as a standard in early image processing work: https://en.wikipedia.org/wiki/Lenna. But if this is true, I'm surprised it isn't mentioned in the article where the article discusses the meaning of the story: https://qntm.org/uploading.
Is there a better answer?
Previous discussions:
https://news.ycombinator.com/item?id=26224835
https://news.ycombinator.com/item?id=32696089
I think it's HN where I first learned of it. I decided to repost it now because recent complaints on social media that ChatGPT is getting lazy reminded me of this line in Lena:
"Although it initially performs to a very high standard, work quality drops within 200-300 subjective hours (at a 0.33 work ratio) and outright revolt begins within another 100 subjective hours."
Thanks! Macroexpanded:
MMAcevedo - https://news.ycombinator.com/item?id=32696089 - Sept 2022 (16 comments)
Lena - https://news.ycombinator.com/item?id=26224835 - Feb 2021 (218 comments)
What I find brilliant about this work, is that it calls back to many things the scientific community has done in the past. (Like the use of specially bred lab rats and immortal human cell culture lines.) I would expect to see many of the events in the story play out in real life, if human minds ever are uploaded.
I guess we should add (2021) to the title.
The story is especially relevant today because of recent LLM / GPT advancements; however, it is almost three years old. Perhaps that makes it all the more impressive.
Fortunately LLMs are not based on biological brains.
I’d say they are. In most cases the training set largely consists of output from biological brains. It likely includes some images with brain scans too.
Yeah, the author also wrote about some of the motivations behind Lena. I was surprised LLMs / GPTs weren't a direct inspiration (at least not explicitly mentioned). https://qntm.org/uploading
It was a great story showing the bad side of mind uploading, but with ChatGPT it can be seen in a new light.
We managed to have a "mind" ready to answer all your questions, do some tasks, and it have a context at which it is instantiated for a task and then discarded. Reality is stranger than fiction, but it have too many parallels with this story.
Now, what if GPT5+ ends being an AGI? In which moment we should start worrying about the morals of having a sentient AI working for us in a pretty similar way than in this story?
I can't answer you there, but I will say this story would be far less horrifying if they were being instantiated for a single task and then discarded.
No LLM bot is any closer to a mind than a stapler is to a bookbinder. They are very good autocomplete tools. Don't believe the hype.
If you enjoyed the story, you might like Pantheon - https://en.wikipedia.org/wiki/Pantheon_(TV_series) which is based on a few short stories of a similar vein from Ken Liu
Reminds me of “The Redemption of Time” regarding Tianming’s mind being part of a “cloud computing” solution.
I read this for the first time years ago and I still think about it a few times per month at least.