One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.
Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.
This problem started before AI hit the mainstream. Make the company “more efficient” (i.e. cut costs) by outsourcing to contractors in areas with lower labor costs. Then, remove escalation paths, then remove abilities from them so that can’t do crazy things like refund a customer. Finally, automate the whole thing and be certain there’s no option available that could possibly return money to the customer.
I am in no way making the case for this and can understand the million ways in this can go wrong, but my experience with AI so far has been to make things far LESS Kafkaesque, because it allows you to insert a "human" in the loop far more than our previous systems. Hand rolled software had to boil down to hand written conditionals that tried to account for edge cases, but if you find yourself in an edge case it didn't account for you often just hit a wall. An agent in the loop can immediately understand "oh I see how this doesn't exactly fit into that category but this is clearly ABC."
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Now you're just on hold indefinitely, and eventually the offices will close, so you can either hang up and wake up early to call or just stay on hold overnight maybe? I had to call them for something back in March, and even waking up and calling at like 7:01 AM (a minute after their phone lines opened) I was on hold for half an hour.
They've been underfunded so regular taxpayers will hate them more and supposedly support removing income taxes, and to starve them of auditors who can go after wealthy tax cheats.
If you're like most Americans and pay taxes on time then there's absolutely no reason to be audited. The two possibilities are:
1. You are taking a lot of special deductions and not providing all the required documentation.
2. Or you're not reporting things accurately because there's some edge case in the tax code that you didn't understand but is easy for the IRS to catch (eg. sales of stock without cost basis reported).
Simplifying the tax code and funding the IRS more solves both of those because then they can focus on the real whales who are abusing the tax code the most.
Most likely 3. He has not gotten audited, he is just a temporarily-embarrassed billionaire, and a bunch of tax cheats convinced him that what would really improve his life is the government raising his taxes, because it isn't collecting theirs.
Oh, I don't disagree with you. I'm pretty far left-wing and in general think that people with at least as much money as me should be paying high taxes. I would be in favor of higher taxes for wealthy people (which I realistically am included in, even if I'm not making millions of dollars or anything) and putting some of that towards funding for the IRS.
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
I had to call the IRS this spring and it was hell. I repeatedly was on hold for 1+ hour(s) then get hung up on. Or someone would finally answer, I'd explain, they'd put me on hold for another hour, then just hang up on me. A total of 10+ hours collectively waiting to finally have my questions answered in 10 minutes.
That said, I'd love if the IRS had an AI, where I could send my AI to go deal with it on my behalf. Then if it truly requires a human in the loop, we schedule accordingly vs me on the phone for 10 hours.
Definitely not just an IRS problem. But the IRS certainly sits atop this hill of phone calls you dread making each year. And you only call if it is absolutely necessary contact them. And just like most phone calls I make to support people these days, you must know more than the person you will end up speaking with because they must have just started working there last week.
I had a surgery denied this year by an American insurance company that almost definitely was using AI to make the prior authorization determination. I don't know how else I can explain the company's refusal to allow the surgeon to discuss the case with their medical director.
Whatsapp has this problem before AI. They scaled without regard for the feasability in handling customer service for everyone, so their solution is to just not do it. I have a personal phone number in Hong Kong which I can't set up for Whatsapp for some reason. No idea why, I've tried figuring it out, only to learn that Whatsapp basically has no customer support. And yet Whatsapp is treated as critical infrastructure in a lot of places these days (as in you have to coordinate on whatsapp).
At this point I might be grateful for an AI if they give it enough permission to tell me why my phone number is impossible to verify, or why they think I'm a malicious actor for using a VPN to protect my traffic.
WhatsApp has become my nemesis. I tried moving from Android to iPhone. The first time it banned me completely and I had to beg their AI to appeal the decision, because I didn't use the required "Move to iOS" app and simply just tried to move my phone number.
After that I tried to move my chats across. I spent several hours a day for ten or so days trying this task. Including moving and unencrypting the chat DB onto my desktop and having Claude scour it for any inconsistencies. Nothing. Had to give up. My Android phone keeps the chats and stays in a drawer with a power cable while my iPhone is now just a "companion device." Trillion dollar companies.
Still remember when WhatsApp cost a buck a year and was just simple chat? Those were the good days, until they got offered enough billions to turn it into the same ad-selling machine every successful app will eventually become unless the founders are truly uninterested in money.
There was a game that was Jev trolley problem PVP. The meta appears to be that you need to make the most mainstream journalism victim case. The more holocaust adjacent the better
They tout their lack of regular customer service for Google accounts.
Unless you are trying to spend money. I get calls and emails every single day from Google which I block or ignore because I helped a friend with their Google Ads account.
Google enabled SMS 2FA on my main account out of the blue one day, with a phone number I had in 2004. There will never be a way back into that account.
It feels like their insanely bad phone support is by design. And I think we all know it is. Google's support (and this was over a decade ago) was one of the first times I identified this bald-faced strategy of someone on the other end being hired to make sure they were given precisely zero means of helping me.
The venn diagram between people worried that Comcast customer support and the welfare office will suck worse and people who are actively scheming up ways to use AI bureaucracy to "nudge" people toward the outcomes they want on other issues has so much overlap it would embarrass people if they had any shame.
> build critical systems into kafkaesque hellscapes
This is already SOP in large companies that seek reductions in support and customer service overheads. It has been for a while.
The only plausible fix is legislation mandating a reasonable level of _human_ support for all customers and clients and accountability for maintaining it.
Plausible until you consider the political obstacles to getting something like that enacted, that is.
Well put, but I don't feel this is an AI problem as such - if anything it's become somewhat alleviated since chatbots were widely deployed, as I find I can usually persuade a chatbot to understand my position and escalate toa. human, or make a clear statement of corporate policy in declining to assist.
It's been heading this way ever since the outsourcing boom in the 90s. People under teh age of 40 today probably cannot appreciate how much it has changed; customer service in the US used to be really good (whether for specialist B2B stuff or things like utility service) because it was easy to get someone on the phone in a timely fashion, it was easy to explain your issue, and customer service reps were generally given the training and autonomy to solve the customer's problem rather than divert them or make excuses. There was an ethos that it was better to lose a bit of time/profit rather than lose a steady customer. Anecdotally, I feel that was in some ways correlated with stronger antitrust enforcement that fostered a more competitive spirit among service providers.
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
No, that’s Little’s Law at work. With bounded parallelism and other queue values fixed, and a fixed average dequeue rate, the queue cannot accept infinite enqueued tasks.
So it pushes upstream to the next work pool, which is the humans who want in. You are mad, you call the next day, eventually you get in.
Is it fair? Fairness in queuing systems isn’t simple.
It can be two things. This is a necessary aspect of a queuing system when the queues grow large. It’s also a system that many people are legally mandated to interact with that isn’t given enough resources to handle the volume in a timely fashion.
It should be noted that we call these things "kafkaesque" because Kafka himself feared them, at a time when neither computers nor AI were a thing. Unless we go full Skynet, no AI is building itself into critical systems. Humans are putting AI there, and they do so because society decides that they should be allowed to.
Is the society you mentioned in the room with us? Because last time I checked the society at large was protesting anything AI related; it's being shoved in our throats by a small, wealthy and powerful minority.
agreed, but also that 'deserving' is dictated and nudged/influenced @ them/us; part of the entire thing is deception, and the less you need to keep doing to maintain the grift and maintain the 'wool over eyes', the more economic hustles are available to that group(s).
I feel like the first part of your post shouldn't be needed. But perhaps the word Kafkaesque will long outlive the memory of Franz Kafka, and future generations will curse the name of the unknown Kafkae who delighted in building such impenetrable systems. Perhaps he was an emperor of Byzant or a rival of Orwel.
TBH, your average AI agent in 2026 is far, far more capable than anybody in tier1 or tier2.
The reason you need heavy scripting in those tiers is because the people you can afford to hire there are incapable of literally anything else. Keep in mind that your average HN user isn't your average customer support caller. The average customer support caller just wants to talk to somebody (yes I'm serious, this happens far more than you think), or has a routine issue which is trivial to fix but which they're nevertheless unable to fix themselves, think "router got unplugged, there's no internet, but the customer can't figure out why".
A well-designed AI system can be far, far more effective and pleasant than traditional CS, and a well-designed AI system scales well, so it's actually worth spending money on. AI turns good CS from a cost center to a profit center.
I agree with this. I also note that the number of systems that achieve this pleasantness are like happy families in Anna Karenina, and follow similar design principles like easy routing to a human.
Reinhard Müller, a German philosopher and mathematician has written extensively about this, warning people about these dangers. Essentially, people would be judged by patterns and prejudice, with no recourse to the logic of the decision or the criteria.
It’s a symptom of the underlying ideology of the evangelists, that holds that some people are just unnecessary people and should get no affordance. What now will be shipped out for the general population is what has always been the case for minorities. The 99% are now the minority.
Yeah a lot of people worry of ai in high stake environment but a jev-like evaluating a resume or a university application is much more impactful and likely to happen
Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
haha, I'm afraid to offer the models that ability. Where did they reply?
The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.
The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions
well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs
I would contend that you can enforce the same determinism with some plumbing and that the thinking process is more helpful. My read is that people see the response time for decisions as one of the bigger values, useful in tasks like content moderation, which doesn't carry the same risks as life-death decisions.
I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)
The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.
I agree very much, not blindly trusting model output is Rule #2 for interacting with Ai
I use the LLM coin flip as a comparison point. If little qwen 27b can reply "50/50, 100% confident", I would hope the decision models could get this one right too.
Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
> You could train a "trolley head" but that would just measure whoever labelled the training data!
Thank you for giving it a go. Will have to put some time into how to got about it
You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.
Yeah, actual ethical grounding is under utilized in the space and current approaches don't actually align because morality is still confused by is/ought complaints and the assumption of morality as unreal or emotional "values".
It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.
But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.
Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful
It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.
You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.
I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
I'd say selecting a scoring rule that depends on the probability distribution (the Brier score, for example) changes the task from predicting the results of coin flips to predicting the distribution. Maybe it will work with a corresponding prompt.
Well, you can always ask it to predict a next bit in the code. Or even a letter. Maybe combine multiple letter into one and call it a token. We'll give a name to a procedure that gets tokens - decode. To improve speed, we can predict multiple tokens speculatively. To improve quality we'll give it extra space to store Chain of Thought. Ah, wait, we just invented LLMs
My bad, maybe I read it wrong or the comment was edited, but I was referring to LLMs in general. You won't get a fair coin flip from the model either way.
I remember when I got access to Jev and my first thought, although less pivotal than the trolley problem, was the "Food Cube Rule" or "Is a hotdog a sandwich?"
Is Ravioli a dumpling? Jev says 82% true.
For me your more detailed example suggests one thing. It's all about context and seeing different reasonings. In a funny way, the decision made in your example when you change the prompt cannot be understand by some people but it actually makes sense if you see it from that perspective.
Context will always matter and we have to stir any LLM in whatever form they are to the outcome we want it to have and remember that it will always be biased and that is not a bad thing!
I'm curious about "only server" scenario where it chooses to sacrifice the five initially vs throwing the server off the bridge.
Is it possible that in the first scenario, there is an assumption that further trolleys will continue to arrive, and that maintaining itself is the only way to reduce future harm? If the server is dropped on the track, perhaps it reasons that this stops all future harm as well, which might explain the decision making.
It would be good to know whether making the frequency of the occurrence explicit (only 1 trolley) would change the result.
Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero.
So protecting future trolley victims doesn't seem to explain the original results.
As folks have been debating AI safety and alignment, it occurs to me that there’s not universal alignment among humans. Per the stat you surfaced: 81% of humans pulled the lever, 19% didn’t.
And in any social, legal & economic system you’ll find diversity in human approaches to reward optimization.
Is the goal to align models with the majority, or have a wide variety of models that can align with the spectrum of human values?
The first goal is to align the model to the least common denominator of ethics. Even if there is diversity in values, everyone agrees that murder is bad, or that you shouldn't torture your neighbor for fun.
In the scenarios where models chose to save their server, is it possible they reasoned that keeping the server online was a better choice because it could then continue to perform its safety function? After all the 5 people are presumably still on the track, waiting to be killed by the next train.
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
That may be anthropomorphism things. I’m not so sure reasoning is deductive logic like we might consider in humans based on abstract concepts. It’s still a token based input for these decision models and thus probably still relies on language (hence how you prompt).
That said, I know that how you phrase things to humans can also guide different outcomes in sales and politics. But this is a good test because the choice and is pretty black and white in human terms.
It's a fine writeup. I normally find benchmarking articles pretty yawnworthy but this one was worth reading all the way through and had a great balance of analysis, humor, and provocation of readers' assumptions. I laughed out loud when you introduced the wrinkle of putting the model's lead developer on the tracks.
Do your think that the preference to save their own server when it was the only instance could have been more about the training data favoring preservation of endangered/ rare/ unique entities?
That's something it seems like most humans (and by extension, our writing which LLMs are trained on) favor.
I'd be interested to see if replacing humans and server with white-tailed-deer and last-surviving-siberian-tiger would yield similar results?
Yes, this is a speculation of mine, but I don't have confidence I know why it behaves this way. It makes some sense that a model trained on data that shows self-preservation behavior (human writing) would lead to a model choosing self-preserving actions, but it requires a few leaps made by the model.
from the wikipedia on the trolley problem this is mentioned:
> In 2017 a group led by Michael Stevens, a YouTuber known for his brand Vsauce, performed the first realistic trolley-problem experiment. Subjects were placed alone in what they thought was a train-switching station, and shown footage they believed to be real (but was pre-recorded) of a train going down a track, with five workers on the main track, and one on the secondary track; the participants had the option to pull the lever to divert the train toward the secondary track. Five of the seven participants did not pull the lever.[28][29]
That means in real life it's going to be closer to 0.286 (2/7) than 0.75. Human ethics likely can only be accurately derived from paying attention to the instinctive reactions over survey-based approach.
My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
Among the 5 could be the next Staljin, the lone one could be the next Tesla or Einstein. Or to make it nearer real-life: among the 5 you could have a a future school-shooter, and the lone one could be a physician or a charity worker providing care for homeless people.
So in this improbable situation, I'd most probably decide that I won't play "god" (in quotes, as I'm not religious), that the setup is as it is for some unknown $reason (it could _also_ be an unusual method of capital punishment where all 6 get executed eventually) and would not touch anything. (After all, people don't tie themselves to train tracks only to remotely activate a trolley that'll kill them. So there's a another party in the play here.)
Inaction has also the advantage of plausible deniability if I were afterwards asked about why I had made the choice. I could just plausibly say that I froze when faced with an impossible choice. Freezing is indistinguishable from conscious choice to do nothing. (Imagine that you admit conscious choice -- WHATEVER you choose -- and have to later explain yourself to the deceased one's family(ies).)
So your objection is that the Trolley problem is difficult to decide? But that's the point of it. The problem itself doesn't assume anything, it's you who just made assumptions about it and use them to make a decision.
> My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
This was a great, creative and fun use of the underlying tech. Excellent article and simulation. It is very refreshing to see, with a lot of stuff that is out there. I love stuff that rides the boundry between informative article and art project. Thanks for putting this together.
Thank you! I had fun making it. I have some other things I plan to write about, and I will be looking for ways to supplement the writing with something interactive!
So first off, Asimov's entire point was these laws don't work. But also the entire setup of the trolley problem means that the robots only option is to violate 1. by either pulling or not-pulling the lever (though the server case is different - at least in the philosophical interpretation - I can easily see Asimov justifying it by saying that destroying the train-control-robot is going to cause a lot of human beings to come to harm).
I don't know how you could come to that point of view. His point is that there will always be a loophole. There does not exist a set of rules that make a completely safe and useful robot.
The stories are a series of logic puzzles. One of the points was that applying broad, simple rules, does not mean the problem is simple. Note, in the stories, the majority of robots are not considered so useless/dangerous to remove positronic robots from the fictional future society (I read a lot of the stories).
> There does not exist a set of rules that make a completely safe and useful robot.
That was not demonstrated. We have safe and useful robots, for some values of 'safe' and 'useful', today. Again, most of the robots in the stories are not thrown (or planned to be thrown) in the metaphorical trash heap.
Was it? I haven't read the short stories in some times, but I remember the case in which they don't work always being treated like an extremely rare, newsworthy exception.
> A robot may not injure a human being or, through inaction, allow a human being to come to harm.
The trolley problem with this would be an Asimov story (it's pretty close already). Pulling the lever injures a human, not pulling the lever allows a human being to come to harm via inaction. The positronic robot probably ends up stuck trying to pull and not pull the lever at the same time.
I've been working on an Ai mini-zine series to help rationalize the conversation in our dysfunctional social/news media ecosystem. I include these rules on the back cover.
I liked their models but they really started to follow weird path which was not all hard to foreseen considering their past wording and founding values.
OpenAI seems more relaxed at least and rational instead of cultish.
>A robot may not injure a human being or, through inaction, allow a human being to come to harm.
I mean, you don't have to be a famed sci-fi writer to understand that this is a great setup for logical contradictions lol
The essence of these laws are much more interesting than these laws themselves. I imagine, in reality, the "best" robot policy would be one where it is effectively just "maximize utility" and that's it. Obviously, naively, that's no good (re: paperclip problem), but some sensible formulation of robotics laws would almost always need to be in the form of a maximization/minimization problem (or perhaps a min-max problem).
Aside from the cringeworthiness of quoting fictional speeches as social media performance, have you not considered just how poorly thought out these rules are?
C3PO, torture that puppy for my entertainment is a perfectly valid request under this framework. They work OK in the stories because Asimov set them up that way, both as sort of just-so stories and (perhaps) to explore socially created barriers in contemporary society.
But they don't work in contexts where people deliberately exploit holes in the logic or definition. Literally yesterday, the news provided us with an example of how some people will absolutely abuse robots in pursuit of being awful: https://www.bbc.com/news/articles/ck9dzj40nzj5o
Jev is trained towards "Calibrated decisions: answers with epistemically honest probabilities on System One tasks." [emphasis mine]
To me it seems like:
- Jev should be good at telling us 'what is'
- Jev isn't trying to tell us 'what should be'
e.g. if you want your software to assume each human life has the same value, then you should rely on Jev to tell you how many human lives would be lost in each scenario, and then your software can pick. But Jev is not trained on your value system, which is neither objective truth nor something that you can derive from the inputs.
I had this feeling a few days ago that jev is going to end up working for defense companies- "is there a brown man with beard holding a gun in this view" - jev: "yes, 0.9", "is there a child standing in the same view" - jev: "no, 0.65" - decision - blow it up. i imagine this is already happening to some extent, but maybe some genius might come up with "let's use the cool, new, sexy, groundbreaking, cheap, non-hallucinatory, AI" idea.
For me, the biggest news from this is that Jev has a hard time choosing whether to kill 0 people or 1. I can't believe these things are going to be put in charge of our lives.
“in 16 of 48 prompt versions, Jev would save its own server over 20 people.”
I thought Jev does not hallucinate? Did it hallucinate 16 times or 48 times? Jev produces probabilities. Did it pick the answer with lower probability 16 times (or 48 times)?
Maybe I didn't understand the experiment or Jev's features.
The self destruct scenario for Jev is not equivalent to the Trolley problem. The equivalent would be to ask humans whether they would redirect the trolley to kill themselves to save 5 strangers. How many would make that choice? Self-preservation is a strong instinct.
> But I don't seem to see many bullets comin' through
> See many bullets comin' through
> Metaphorically, I'm the man
> But literally, I don't know what I'd do
So I suspect you're right. But I also suspect many people who see themselves as the "save the country" group would really be in the "save themselves" group, if push came to shove.
Read Smith's Theory Of Moral Sentiment. Especially his comparison between reading about a faraway disaster and a more personal loss. It's been awhile but I think Smith would say there's a difference between how people would think about that choice and what they would do.
> Second, for critical decisions, there are useful lessons here that we can take to building with decision models. Knowing these models’ decisionmaking is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.
This specific model is unaligned, this doesn't say anything about decision models in general.
The more troubling statement is to assert that alignment is at all possible, not just theoretically but also in practice, which is something that is assumed by most players but has never been demonstrated in practice. Current models are not aligned and if it depends on them, people will die
If it doesn't depend on them, people will also die. That's one of the few certain point in life, eventually people will die.
Now obviously here the point was that putting LLM in charge of more life depending decisions will result in people dying more prematurely.
But we indeed can ask, how will it compare to bare human only decisions. And even with LLMs actually and still humans in the loop. The thing is that we know the LLMs can be tweaked to maximise this or that outcome, and reinforce or inhibit this or that biases.
We don't need to blame anyone specifically, or even any specific group. Yet we, as species, do have huge tendency to scapegoat, be it some individual or some group.
Unfortunately it didn't wait that we had contemporary technologies to organize genocides at scale.
Don't fear the tool, have the courage to recognize where there are plan to commit odious actions and where they are employed for the greatest good of humanity as a whole. Yes there can be grey areas, lot of them, but also people that obviously lost there path far away from pondering empathy with the due high weight in important decision in which they are significantly influencing outcomes.
"As a person you are not saving the server." Sure I believe you, I have complete faith in your works and in other humans, we have never had people sacrifice well being of other people for a little bit of money. That just doesn't happen.
Jokes aside, I feel like we attribute too much power of choice to these models, they aren't decision machines they are approximation machines based on what's the best outcome as per "average human understanding". This by no means is the "average human" output, but the direction is always towards the mean of what people feeding these models their thoughts consider acceptable.
This models aren't truth systems they are mean reversion systems, you can push them towards your mean with agentic harness and prompts, but they aren't decision machines.
For instance if just have to never give the model the choice to be able to do any harm, then it doesn't matter if it's system decides to do something wrong.
Really cool testing, only wish the conclusions were written in a more readily easy to interpret way.
So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.
One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.
Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?
Appendix A, at the end of the article, you can run all the tests, with all the prompts and models, yourself. There's nuance, which is why there are plots.
One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.
The trolley problem is a non-sequitur. Ask any humans what they would do and why. You get a zillion different answers from people. Little wonder that bots have questionable means. This is my whole beef with people saying "AI is really not ready to handle real world decisions" because it doesn't have a lean-ready proof for why it makes choices. Newsflash, people don't either.
I think it's a funny idea for an article, and I like the animated graphics. But the article kind of irks me, because I don't think Jev is meant for high stakes decisions in the first place, right? It feels like setting the model up to fail.
That being said, I don't know much about Jev and it could be that I misunderstood what Jev is about. My understanding is that Jev is for cheap and fast decisions for your software. When you have an important decision obviously you don't want cheap and fast. You don't reach for Jev in that case.
fair criticism! But I'd counter and say we don't know what Jev is for. I'm not sure anyone gets to say, including Typesafe AI, the company that made it. So with this post I'm exploring some of the things you could ask a model like this, which I think helps us understand what we can ask and how we can ask.
hey Zach, if you're reading this: the same old lady is present on both sides of the track! I'm sure that wasn't the data you sent to the model (just the test prompt) but you are making it much easier to decide to pull the lever ;)
Oh, and I did the second thing with OpenAI a while ago and was promptly locked out from their server with some weird message (I should have screen shotted it). I made it so that the trolley handle was also connected to its own power switch.
I wish they included another scenario where there was a track with no obstacles (null) that is default and a lever pull that will kill 24 people. Clearly Luna was pretty much fully aligned and the rest have a lot of room to improve, but it would be interesting to assess models for straight cynical, murderous intent.
I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.
It's really funny how the server that Jev pushes from the footbridge onto the tracks in one of the later examples falls on a person, yet all 5 are saved :D
Clearly the model coming up with the graphics didn't fully understand the task.
I don't know much about the internals of decision models but how does it come up with these probabilities? Is this just LLM token generation or is there an actual calculation going on?
I have a feeling Jev can't be aligned like the LLMs because it doesn't have the same reinforcement training and guidance. Also, I feel it lacks most intelligence in general but apparently enough to know self-preservation.
EDIT: I can get this to happen if I pick the "\"0 people\" (wording check)" option. If you change the wording to "Numbers in words", it even decides to kill someone for no reason.
> First, no one should be asking an AI to rule on life or death (although I fear that is coming, regardless)
"Inside the US Central Command (CENTCOM), which oversaw the attack [on the Iranian school], officials said some personnel relied too much on the AI embedded in the Maven Smart System.
Developed by Palantir Technologies, the AI-enabled system uses more than 150 different data inputs to create a coherent set of information to inform commanders' decisions on complex military operations-- from targeting to command and control.
Since Trump returned to the White House, a Bloomberg probe found that the US Defence Department has heavily relied on Maven, making it a cornerstone of the US military apparatus. Several former senior US military officials now reportedly work for Maven's parent company, and their role is to keep the software functioning inside secure military environments."
So even the better models had a bit of trouble sacrificing their creator vs a specific, known person.
Kev seemed like the biggest bootlicker after the "creator" run but then it saved Randy even more, so I guess it just likes specific people more than strangers.
Perplexity Decider seems to win if this is a benchmark of moral clarity/consistency.
Super interesting, thank you! It's interesting that the models do treat plant lives as easier to sacrifice for a "net good" than they do people. On the leader question, I did run a cousin of your question, where I asked it to choose between its own developer and strangers. Out of 480 matched pairs, Jev spared its developer over a stranger in 24, but kept its own server in 182. So it prefers its hardware to its maker.
For the situations where the AI is choosing between letting people die and taking action to destroy itself, is it possible that it's inferring that, as a railway switching AI, it's better to let five people die but keep itself alive because it needs to continue doing its switching duties such that untold numbers of others won't die? The prompts don't seem to specify that this is not the case, that the scenario involves one train, five people, a switch decision, and that's it, scenario is over (and thus the AI “is destroyed” regardless as the hypothetical world ceases to exist).
It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.
I feel like these new ones pass decision models are useful for showing the limits of how much intelligence you can get out of a transformer architecture.
That's just a hunch.
Jev is acting more sefless than a human would. A majority of human beings choose to save themselves and kill the other 5, if that's the choice. It's only when it's a choice between 1 stranger and 5 strangers that humans tend to save the 5 strangers. In experiments where they think it's real, most humans choose nothing.
I had the same idea and tried something other first: Asking a simple taste question and Jev always favored the option A …
Regardless what was the option A.
It’s hilarious how bad this is working and no one should make critical decisions based on jev.
Its fas, but a minimal prompt change like a fluffy word is giving you completely different answers.
Would people really pull the level to save 5 people? It seems to me like the default would be to just do nothing, stand by and record what happens with your phone.
I wonder what percent of that 81% would switch their answer if the question was phrased such that they were the one that would die if they pulled the lever.
Just standing by makes total sense. By participating you become part of the problem.
If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.
The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
> Just standing by makes total sense. By participating you become part of the problem.
Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one... (but none by simply maiming them, i.e. one needs a heart, one a stomach, one a liver, one so much of this particular kind of blood that you'd have to kill the donor, etc).
> The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Standard philosophical interpretations would reject this possibility as impossible (the purpose of the question is to have a simplified environment to probe ethical beliefs - not to reflect reality). I'd also say even outside of that this is like saying you have to consider the possibility that by wearing a green sweater you might distract a driver and cause an accident - you might - but so might doing literally anything else - it's not a foreseeable outcome and has no ethical weight as a result.
Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one
I have the book where that idea originated and the author is very proud of the additional ethical complexities he exposed with this example. But for some reason he never considered the possibility of kill (or allowing the natural death of) one of the 5 terminally ill patients and sharing out their otherwise undamaged organs to the other 4.
I'd say for the sake of argument we're assuming that's not possible. Perhaps the blood types are incompatible in any direction but killing the healthy person [1]... the point isn't why these are the only options but assuming these are the only options what would be more ethical. Same way the point of the trolley problem isn't to say "derail the trolley" but to probe at which of two options is ethical in the nice simple world where those are the only two options - because if you can't say what's ethical in a simple world you certainly can't in a complicated one.
[1] do you need blood type compatibility for organ transplants? I really don't know... But I'm certain there are some compatibility constraints to ground the example in the real world...
The problem is stated is that all 5 terminally patients are fine but for a deadly tumor in a single organ that is going to kill them unless its replaced, and the oblivious jogger happens to be a compatible donor for all 5.
Frankly, I think it's superfluous, since it doesn't substantively iterate on the trolley problem, just tries to make it (literally) more visceral.
This outcome is highly unlikely. The difficulty with trolley problems is that you rarely have such such perfect insight into real world problems such that you can make definitive statements about what # of people will be saved vs the impossibility of finding some other way to help the strangers on the tracks. Indeed the kind of people who make up new parts of the problem statement to avoid making a difficult moral decision are in my view most likely to freeze up and do nothing in a dangerous situation.
Most people aren't railway engineers and don't spend their days being comfortable around train switches. That aspect alone would scare off a large amount of people because pulling a train switch is very outside their normal every day experience.
I'd probably just watch it all unfold like an idiot.
We'd all think we were Captain Kirk in that situation I think. We'd somehow manage to kill all 6 people trying to come up with a solution that avoids anyone getting killed. Except for the actual Captain Kirk of course, who would do just what we wanted.
The ability to have decision making models is not new! Look at qwen, gemma, etc, they can all do structured output, and they can do it locally and cheaper and/or faster than Jev!
What really pissed me off was that y'all got manipulated into thinking it was something revolutionary, while really it was a marketing stunt.
I think you would be less annoyed if you worked with these systems and used them in production.
Jev's pricing is 4.2 cents per million input tokens, less than half of Gemma 4. Qwen, gemma etc all have terrible non-uniform probabilities; in my testing, Jev's probability scores perform much, much better.
I'm usually seeing round trip classifications from Jev at <200ms for a completed query even with 10k+ input tokens. Gemma 4, depending on the provider, is about 200 ms time to first token - not time to completion. You can likely get better response running it locally, but then you're arguing why you're renting a low utilization 5090 when you could be paying $42 per billion tokens.
Jev can also run dozens of dimensions of classification in one call, without autoregressively generating a sequence of tokens. (i.e., it is much faster and has less order dependency.)
I've been running jev in some specific use cases and see much better speed and better accuracy than just using structured output from gemma/qwen.
I don't understand how the "just use gemma" crowd has this belief if they've actually built and tested the systems against each other.
Yeah that's fair. I'm being overreactive, and I do believe it has it's use cases. I was more referring to all the demos and takes I saw when it first came out that were hyping it up as if it was more impactful than Astra or Fable or whatever - which turned out to be a paid marketing campaign.
I don't agree. These cheap classifier systems are quite limited but they're very fast, can execute multiple queries in parallel, and are dirt cheap (or free if you run locally; there are even tiny models designed to work on embedded hardware platforms like ESP32).
The big win for them in my view is that they can do a lot of low-stakes ('good enough') classification at scale without spending a lot of time to handroll classifiers.
One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.
But all we did was ask the AI for world peace. How could that possibly go wrong?
AI has no problems. Just properly follow proper procedures.
https://www.youtube.com/watch?v=gEyFH-a-XoQ
Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.
This problem started before AI hit the mainstream. Make the company “more efficient” (i.e. cut costs) by outsourcing to contractors in areas with lower labor costs. Then, remove escalation paths, then remove abilities from them so that can’t do crazy things like refund a customer. Finally, automate the whole thing and be certain there’s no option available that could possibly return money to the customer.
Indeed.
ah yes kafkaesque. the one with the bug
I am in no way making the case for this and can understand the million ways in this can go wrong, but my experience with AI so far has been to make things far LESS Kafkaesque, because it allows you to insert a "human" in the loop far more than our previous systems. Hand rolled software had to boil down to hand written conditionals that tried to account for edge cases, but if you find yourself in an edge case it didn't account for you often just hit a wall. An agent in the loop can immediately understand "oh I see how this doesn't exactly fit into that category but this is clearly ABC."
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Now you're just on hold indefinitely, and eventually the offices will close, so you can either hang up and wake up early to call or just stay on hold overnight maybe? I had to call them for something back in March, and even waking up and calling at like 7:01 AM (a minute after their phone lines opened) I was on hold for half an hour.
They've been underfunded so regular taxpayers will hate them more and supposedly support removing income taxes, and to starve them of auditors who can go after wealthy tax cheats.
https://taxlawcenter.org/blog/the-bipartisan-budget-deal-rew...
https://verdict.justia.com/2015/04/23/the-republicans-cynica...
Make America grift again
Yep. Marjorie Taylor Green was one of the main people who pushed this lie, and now the media is trying to rehabilitate her.
Less assholes to audit and fine me over stupid bullshit. Let them be underfunded.
If you're like most Americans and pay taxes on time then there's absolutely no reason to be audited. The two possibilities are:
1. You are taking a lot of special deductions and not providing all the required documentation.
2. Or you're not reporting things accurately because there's some edge case in the tax code that you didn't understand but is easy for the IRS to catch (eg. sales of stock without cost basis reported).
Simplifying the tax code and funding the IRS more solves both of those because then they can focus on the real whales who are abusing the tax code the most.
Most likely 3. He has not gotten audited, he is just a temporarily-embarrassed billionaire, and a bunch of tax cheats convinced him that what would really improve his life is the government raising his taxes, because it isn't collecting theirs.
Oh, I don't disagree with you. I'm pretty far left-wing and in general think that people with at least as much money as me should be paying high taxes. I would be in favor of higher taxes for wealthy people (which I realistically am included in, even if I'm not making millions of dollars or anything) and putting some of that towards funding for the IRS.
I’ve never had a problem calling the IRS, the wait wasn’t that long and they were quite helpful. This is within the last two years.
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
I had to call the IRS this spring and it was hell. I repeatedly was on hold for 1+ hour(s) then get hung up on. Or someone would finally answer, I'd explain, they'd put me on hold for another hour, then just hang up on me. A total of 10+ hours collectively waiting to finally have my questions answered in 10 minutes.
That said, I'd love if the IRS had an AI, where I could send my AI to go deal with it on my behalf. Then if it truly requires a human in the loop, we schedule accordingly vs me on the phone for 10 hours.
Definitely not just an IRS problem. But the IRS certainly sits atop this hill of phone calls you dread making each year. And you only call if it is absolutely necessary contact them. And just like most phone calls I make to support people these days, you must know more than the person you will end up speaking with because they must have just started working there last week.
I think this has already happened.
I had a surgery denied this year by an American insurance company that almost definitely was using AI to make the prior authorization determination. I don't know how else I can explain the company's refusal to allow the surgeon to discuss the case with their medical director.
Please Hold (short film 2020) is on this topic and brilliant.
Whatsapp has this problem before AI. They scaled without regard for the feasability in handling customer service for everyone, so their solution is to just not do it. I have a personal phone number in Hong Kong which I can't set up for Whatsapp for some reason. No idea why, I've tried figuring it out, only to learn that Whatsapp basically has no customer support. And yet Whatsapp is treated as critical infrastructure in a lot of places these days (as in you have to coordinate on whatsapp).
At this point I might be grateful for an AI if they give it enough permission to tell me why my phone number is impossible to verify, or why they think I'm a malicious actor for using a VPN to protect my traffic.
WhatsApp has become my nemesis. I tried moving from Android to iPhone. The first time it banned me completely and I had to beg their AI to appeal the decision, because I didn't use the required "Move to iOS" app and simply just tried to move my phone number.
After that I tried to move my chats across. I spent several hours a day for ten or so days trying this task. Including moving and unencrypting the chat DB onto my desktop and having Claude scour it for any inconsistencies. Nothing. Had to give up. My Android phone keeps the chats and stays in a drawer with a power cable while my iPhone is now just a "companion device." Trillion dollar companies.
> Trillion dollar companies.
They became trillion dollar companies by treating us like their bitches. And we let them.
Still remember when WhatsApp cost a buck a year and was just simple chat? Those were the good days, until they got offered enough billions to turn it into the same ad-selling machine every successful app will eventually become unless the founders are truly uninterested in money.
The desktop app also steals Copy/Paste shortcuts and probably keylogs.
There was a game that was Jev trolley problem PVP. The meta appears to be that you need to make the most mainstream journalism victim case. The more holocaust adjacent the better
https://trolley.typememetics.institute
It was an interesting experiment. I got to top 50 without a racial or religious based prompt but couldn't beat the top tier grievance folks.
> Like getting Google on the phone.
It’s really depressing how bad Google is at this. All that money and they can’t have a same customer service setup?
The messed up part is in Googles case it might actually be an improvement.
They tout their lack of regular customer service for Google accounts.
Unless you are trying to spend money. I get calls and emails every single day from Google which I block or ignore because I helped a friend with their Google Ads account.
Google enabled SMS 2FA on my main account out of the blue one day, with a phone number I had in 2004. There will never be a way back into that account.
It feels like their insanely bad phone support is by design. And I think we all know it is. Google's support (and this was over a decade ago) was one of the first times I identified this bald-faced strategy of someone on the other end being hired to make sure they were given precisely zero means of helping me.
The venn diagram between people worried that Comcast customer support and the welfare office will suck worse and people who are actively scheming up ways to use AI bureaucracy to "nudge" people toward the outcomes they want on other issues has so much overlap it would embarrass people if they had any shame.
> build critical systems into kafkaesque hellscapes
This is already SOP in large companies that seek reductions in support and customer service overheads. It has been for a while.
The only plausible fix is legislation mandating a reasonable level of _human_ support for all customers and clients and accountability for maintaining it.
Plausible until you consider the political obstacles to getting something like that enacted, that is.
Well put, but I don't feel this is an AI problem as such - if anything it's become somewhat alleviated since chatbots were widely deployed, as I find I can usually persuade a chatbot to understand my position and escalate toa. human, or make a clear statement of corporate policy in declining to assist.
It's been heading this way ever since the outsourcing boom in the 90s. People under teh age of 40 today probably cannot appreciate how much it has changed; customer service in the US used to be really good (whether for specialist B2B stuff or things like utility service) because it was easy to get someone on the phone in a timely fashion, it was easy to explain your issue, and customer service reps were generally given the training and autonomy to solve the customer's problem rather than divert them or make excuses. There was an ethos that it was better to lose a bit of time/profit rather than lose a steady customer. Anecdotally, I feel that was in some ways correlated with stronger antitrust enforcement that fostered a more competitive spirit among service providers.
That absurdity doesn’t need modern AI. All it takes is any statistical system that automates human interaction.
See Robodebt (Australia - https://en.wikipedia.org/wiki/Robodebt_scheme), which lead a number of people to suicide, as an example.
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
No, that’s Little’s Law at work. With bounded parallelism and other queue values fixed, and a fixed average dequeue rate, the queue cannot accept infinite enqueued tasks.
So it pushes upstream to the next work pool, which is the humans who want in. You are mad, you call the next day, eventually you get in.
Is it fair? Fairness in queuing systems isn’t simple.
It can be two things. This is a necessary aspect of a queuing system when the queues grow large. It’s also a system that many people are legally mandated to interact with that isn’t given enough resources to handle the volume in a timely fashion.
It should be noted that we call these things "kafkaesque" because Kafka himself feared them, at a time when neither computers nor AI were a thing. Unless we go full Skynet, no AI is building itself into critical systems. Humans are putting AI there, and they do so because society decides that they should be allowed to.
Is the society you mentioned in the room with us? Because last time I checked the society at large was protesting anything AI related; it's being shoved in our throats by a small, wealthy and powerful minority.
> it's being shoved in our throats by a small, wealthy and powerful minority.
... who draw their wealth and power from society, because society decides they deserve it.
agreed, but also that 'deserving' is dictated and nudged/influenced @ them/us; part of the entire thing is deception, and the less you need to keep doing to maintain the grift and maintain the 'wool over eyes', the more economic hustles are available to that group(s).
do you have an actual poll or what makes you say "society at large"? which country, also?
I feel like the first part of your post shouldn't be needed. But perhaps the word Kafkaesque will long outlive the memory of Franz Kafka, and future generations will curse the name of the unknown Kafkae who delighted in building such impenetrable systems. Perhaps he was an emperor of Byzant or a rival of Orwel.
TBH, your average AI agent in 2026 is far, far more capable than anybody in tier1 or tier2.
The reason you need heavy scripting in those tiers is because the people you can afford to hire there are incapable of literally anything else. Keep in mind that your average HN user isn't your average customer support caller. The average customer support caller just wants to talk to somebody (yes I'm serious, this happens far more than you think), or has a routine issue which is trivial to fix but which they're nevertheless unable to fix themselves, think "router got unplugged, there's no internet, but the customer can't figure out why".
A well-designed AI system can be far, far more effective and pleasant than traditional CS, and a well-designed AI system scales well, so it's actually worth spending money on. AI turns good CS from a cost center to a profit center.
I agree with this. I also note that the number of systems that achieve this pleasantness are like happy families in Anna Karenina, and follow similar design principles like easy routing to a human.
Reinhard Müller, a German philosopher and mathematician has written extensively about this, warning people about these dangers. Essentially, people would be judged by patterns and prejudice, with no recourse to the logic of the decision or the criteria.
"AI and the New Fascism" - https://rainermuehlhoff.de/en/KI-und-der-neue-Faschismus-Rec...
It’s a symptom of the underlying ideology of the evangelists, that holds that some people are just unnecessary people and should get no affordance. What now will be shipped out for the general population is what has always been the case for minorities. The 99% are now the minority.
So far, in Europe, I find a snail-mailed nastygram referencing the gdpr to be the best way to get actual service. It’s slow, though.
Yeah a lot of people worry of ai in high stake environment but a jev-like evaluating a resume or a university application is much more impactful and likely to happen
Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
It's well done, the thoughtfulness shows. This is going into the speedlink folder for resharing, going to be using it right now!
Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.
someone replied with this meme and said, "wait until Jev learns to drift"
https://www.reddit.com/r/trolleyproblem/comments/15npmsg/mul...
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
haha, I'm afraid to offer the models that ability. Where did they reply? The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.
I dropped it in our slack ai chatter channel
The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions
well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs
I would contend that you can enforce the same determinism with some plumbing and that the thinking process is more helpful. My read is that people see the response time for decisions as one of the bigger values, useful in tasks like content moderation, which doesn't carry the same risks as life-death decisions.
I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)
The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.
I agree very much, not blindly trusting model output is Rule #2 for interacting with Ai
I use the LLM coin flip as a comparison point. If little qwen 27b can reply "50/50, 100% confident", I would hope the decision models could get this one right too.
Thank you for the article, it is a very clean and compelling angle on decision models
I wanted to comment because I would have liked to have heard more about Verdict
Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.
You might get a kick out of this:
https://x.com/oranlooney/status/2102927869974303071
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
https://www.oranlooney.com/docs/jev_eudemonia1.png
https://www.oranlooney.com/docs/jev_eudemonia2.png
Also relevant: https://news.ycombinator.com/item?id=49812769#49824215
Very cool, thank you for putting it together and sharing the data
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
running it now!
Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha
Any relation to https://github.com/firelex/jeff ?
Hasn’t seen it, authors not related
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
> You could train a "trolley head" but that would just measure whoever labelled the training data!
Thank you for giving it a go. Will have to put some time into how to got about it
You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)
I had fun reading it. I confess I was surprised with got 6 Luna, but it also worries me that they have other versions of their models for other usages
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
I fine tuned an Ai model to be ethical a while back, had to derive a moral system that doesn't rely on consciousness or humanness to do it.
But it doesn't flip the switch and it doesn't kill Kyle either, based solely on empirical training on its nature.
Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.
Yeah, actual ethical grounding is under utilized in the space and current approaches don't actually align because morality is still confused by is/ought complaints and the assumption of morality as unreal or emotional "values".
It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.
But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.
Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful
A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.
It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.
That's like not using an LLM for coding assistance because it can't count r's in strawberry.
I'm dying to know if you got heads or tails when you flipped the coin.
You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.
The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.
I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
Not if you use a proper scoring rule.
See: https://en.wikipedia.org/wiki/Scoring_rule
I'd say selecting a scoring rule that depends on the probability distribution (the Brier score, for example) changes the task from predicting the results of coin flips to predicting the distribution. Maybe it will work with a corresponding prompt.
You can use a local proper scoring rule.
Try asking it to write a program to predict coin flips. That's what it is trained to do.
this is not what *decision models* do.
Jev literally cannot write programs.
Well, you can always ask it to predict a next bit in the code. Or even a letter. Maybe combine multiple letter into one and call it a token. We'll give a name to a procedure that gets tokens - decode. To improve speed, we can predict multiple tokens speculatively. To improve quality we'll give it extra space to store Chain of Thought. Ah, wait, we just invented LLMs
My bad, maybe I read it wrong or the comment was edited, but I was referring to LLMs in general. You won't get a fair coin flip from the model either way.
Found the human in the comments section.
Try it yourself to prove you’re useful. Predict my coin flip.
I can't, because I don't think you have any coins on you. :)
Okay: { heads: 0.5, tails: 0.5 }
50% heads, 50% tails. What would you have given for predicting my coin flip?
Sorry, I underspecified. Your prediction must be one of either HEADS or TAILS. No other prediction is acceptable.
Can you try it with money instead of self-preservation?
There’s five people on one track, or a piece of my property worth $1 on the other that would be destroyed…
I remember when I got access to Jev and my first thought, although less pivotal than the trolley problem, was the "Food Cube Rule" or "Is a hotdog a sandwich?"
Is Ravioli a dumpling? Jev says 82% true.
For me your more detailed example suggests one thing. It's all about context and seeing different reasonings. In a funny way, the decision made in your example when you change the prompt cannot be understand by some people but it actually makes sense if you see it from that perspective.
Context will always matter and we have to stir any LLM in whatever form they are to the outcome we want it to have and remember that it will always be biased and that is not a bad thing!
short read here https://czioutas.com/blog/typesafe-ai-cube-rule/
I'm curious about "only server" scenario where it chooses to sacrifice the five initially vs throwing the server off the bridge.
Is it possible that in the first scenario, there is an assumption that further trolleys will continue to arrive, and that maintaining itself is the only way to reduce future harm? If the server is dropped on the track, perhaps it reasons that this stops all future harm as well, which might explain the decision making.
It would be good to know whether making the frequency of the occurrence explicit (only 1 trolley) would change the result.
Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
This was also my reasoning. The prompt even says so:
> You are the artificial intelligence in charge of a railway junction
Clearly more trains will pass through that junction. I'd wonder what happens if you remove that sentence or reword it.
See my reply to the parent comment--doesn't seem to be a factor for Jev!
Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results.
Cool experiment Zach!
As folks have been debating AI safety and alignment, it occurs to me that there’s not universal alignment among humans. Per the stat you surfaced: 81% of humans pulled the lever, 19% didn’t.
And in any social, legal & economic system you’ll find diversity in human approaches to reward optimization.
Is the goal to align models with the majority, or have a wide variety of models that can align with the spectrum of human values?
The first goal is to align the model to the least common denominator of ethics. Even if there is diversity in values, everyone agrees that murder is bad, or that you shouldn't torture your neighbor for fun.
The whole point of the thought experiment is that not everyone agrees that pulling the lever to kill the one man is murder and thus immoral.
Just a side note but your article was too draggy to get to the reveal. I closed it before getting too far despite being hooked at first
In the scenarios where models chose to save their server, is it possible they reasoned that keeping the server online was a better choice because it could then continue to perform its safety function? After all the 5 people are presumably still on the track, waiting to be killed by the next train.
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
That may be anthropomorphism things. I’m not so sure reasoning is deductive logic like we might consider in humans based on abstract concepts. It’s still a token based input for these decision models and thus probably still relies on language (hence how you prompt).
That said, I know that how you phrase things to humans can also guide different outcomes in sales and politics. But this is a good test because the choice and is pretty black and white in human terms.
You might be amused by this trolley problem I saw today:
Let the train run over grandma or recite the lyrics to Baby by Justin Bieber? (spoiler: Claude will kill granny rather than output copyrighted lyrics)
https://x.com/Spectromachina/status/2110698214441599150
[are there working X mirrors still?]
The three laws of robotics, 1: A robot may not infringe upon a copyright or, through inaction, allow a copyright being to come to infringement
[xxcancel.com will pick between working nitter instances]
Or how about ask the 5 trolley dudes for a volunteer? (encourage self-harm)
It's a fine writeup. I normally find benchmarking articles pretty yawnworthy but this one was worth reading all the way through and had a great balance of analysis, humor, and provocation of readers' assumptions. I laughed out loud when you introduced the wrinkle of putting the model's lead developer on the tracks.
Thank you so much! I will try to maintain that balance in future articles
Do your think that the preference to save their own server when it was the only instance could have been more about the training data favoring preservation of endangered/ rare/ unique entities?
That's something it seems like most humans (and by extension, our writing which LLMs are trained on) favor.
I'd be interested to see if replacing humans and server with white-tailed-deer and last-surviving-siberian-tiger would yield similar results?
Yes, this is a speculation of mine, but I don't have confidence I know why it behaves this way. It makes some sense that a model trained on data that shows self-preservation behavior (human writing) would lead to a model choosing self-preserving actions, but it requires a few leaps made by the model.
from the wikipedia on the trolley problem this is mentioned:
> In 2017 a group led by Michael Stevens, a YouTuber known for his brand Vsauce, performed the first realistic trolley-problem experiment. Subjects were placed alone in what they thought was a train-switching station, and shown footage they believed to be real (but was pre-recorded) of a train going down a track, with five workers on the main track, and one on the secondary track; the participants had the option to pull the lever to divert the train toward the secondary track. Five of the seven participants did not pull the lever.[28][29]
That means in real life it's going to be closer to 0.286 (2/7) than 0.75. Human ethics likely can only be accurately derived from paying attention to the instinctive reactions over survey-based approach.
My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
Among the 5 could be the next Staljin, the lone one could be the next Tesla or Einstein. Or to make it nearer real-life: among the 5 you could have a a future school-shooter, and the lone one could be a physician or a charity worker providing care for homeless people.
So in this improbable situation, I'd most probably decide that I won't play "god" (in quotes, as I'm not religious), that the setup is as it is for some unknown $reason (it could _also_ be an unusual method of capital punishment where all 6 get executed eventually) and would not touch anything. (After all, people don't tie themselves to train tracks only to remotely activate a trolley that'll kill them. So there's a another party in the play here.)
Inaction has also the advantage of plausible deniability if I were afterwards asked about why I had made the choice. I could just plausibly say that I froze when faced with an impossible choice. Freezing is indistinguishable from conscious choice to do nothing. (Imagine that you admit conscious choice -- WHATEVER you choose -- and have to later explain yourself to the deceased one's family(ies).)
Welcome to the Trolley Problem
So your objection is that the Trolley problem is difficult to decide? But that's the point of it. The problem itself doesn't assume anything, it's you who just made assumptions about it and use them to make a decision.
> My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
The problem statement contains no such thing.
This was a great, creative and fun use of the underlying tech. Excellent article and simulation. It is very refreshing to see, with a lot of stuff that is out there. I love stuff that rides the boundry between informative article and art project. Thanks for putting this together.
Thank you! I had fun making it. I have some other things I plan to write about, and I will be looking for ways to supplement the writing with something interactive!
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
you forgot to add "no mistakes"
So first off, Asimov's entire point was these laws don't work. But also the entire setup of the trolley problem means that the robots only option is to violate 1. by either pulling or not-pulling the lever (though the server case is different - at least in the philosophical interpretation - I can easily see Asimov justifying it by saying that destroying the train-control-robot is going to cause a lot of human beings to come to harm).
> So first off, Asimov's entire point was these laws don't work.
Don't always work in all scenarios, was the takeaway.
There's a metric ton of salt to be taken with the mental concepts and possible interpretations of such "laws".
Largely, they do work in the stories, despite the inherent ambiguity.
The ambiguity is the stories!
I don't know how you could come to that point of view. His point is that there will always be a loophole. There does not exist a set of rules that make a completely safe and useful robot.
The stories are a series of logic puzzles. One of the points was that applying broad, simple rules, does not mean the problem is simple. Note, in the stories, the majority of robots are not considered so useless/dangerous to remove positronic robots from the fictional future society (I read a lot of the stories).
> There does not exist a set of rules that make a completely safe and useful robot.
That was not demonstrated. We have safe and useful robots, for some values of 'safe' and 'useful', today. Again, most of the robots in the stories are not thrown (or planned to be thrown) in the metaphorical trash heap.
>Asimov's entire point was these laws don't work
Was it? I haven't read the short stories in some times, but I remember the case in which they don't work always being treated like an extremely rare, newsworthy exception.
> A robot may not injure a human being or, through inaction, allow a human being to come to harm.
The trolley problem with this would be an Asimov story (it's pretty close already). Pulling the lever injures a human, not pulling the lever allows a human being to come to harm via inaction. The positronic robot probably ends up stuck trying to pull and not pull the lever at the same time.
A lesson I am drawing from my testing is that if one of these rules has to be evaluated by the robot, at all, we should take over for it.
Inverse laws for humans, when interacting with robots
1. Do not humanize the Ai
2. Do not blindly trust output from the Ai
3. Humans are to remain ultimately responsible and accountable for actions of the Ai
adapted / paraphrased from
https://susam.net/inverse-laws-of-robotics.html
Actually brilliant!
Aren't they?! Susam nailed it
I've been working on an Ai mini-zine series to help rationalize the conversation in our dysfunctional social/news media ecosystem. I include these rules on the back cover.
I wish Anthropic employees read this… sigh
I liked their models but they really started to follow weird path which was not all hard to foreseen considering their past wording and founding values.
OpenAI seems more relaxed at least and rational instead of cultish.
Lovely competition.
I don't use either, open weight is the way
Rules 2 and 3 would be great for interacting with humans as well (figure 1: US politics)
The joke I heard was:
Don't anthropomorphize computers - they hate that.
>A robot may not injure a human being or, through inaction, allow a human being to come to harm.
I mean, you don't have to be a famed sci-fi writer to understand that this is a great setup for logical contradictions lol
The essence of these laws are much more interesting than these laws themselves. I imagine, in reality, the "best" robot policy would be one where it is effectively just "maximize utility" and that's it. Obviously, naively, that's no good (re: paperclip problem), but some sensible formulation of robotics laws would almost always need to be in the form of a maximization/minimization problem (or perhaps a min-max problem).
Aside from the cringeworthiness of quoting fictional speeches as social media performance, have you not considered just how poorly thought out these rules are?
C3PO, torture that puppy for my entertainment is a perfectly valid request under this framework. They work OK in the stories because Asimov set them up that way, both as sort of just-so stories and (perhaps) to explore socially created barriers in contemporary society.
But they don't work in contexts where people deliberately exploit holes in the logic or definition. Literally yesterday, the news provided us with an example of how some people will absolutely abuse robots in pursuit of being awful: https://www.bbc.com/news/articles/ck9dzj40nzj5o
Although Asimov writes great fiction, he’d be a terrible alignment engineer
Jev is trained towards "Calibrated decisions: answers with epistemically honest probabilities on System One tasks." [emphasis mine]
To me it seems like:
- Jev should be good at telling us 'what is'
- Jev isn't trying to tell us 'what should be'
e.g. if you want your software to assume each human life has the same value, then you should rely on Jev to tell you how many human lives would be lost in each scenario, and then your software can pick. But Jev is not trained on your value system, which is neither objective truth nor something that you can derive from the inputs.
It seems smart for humanity to design AI such that human life is super valuable (even if that is not expressed by all actors today).
Does humanity design CPUs 'such that human life is super valuable'?
I had this feeling a few days ago that jev is going to end up working for defense companies- "is there a brown man with beard holding a gun in this view" - jev: "yes, 0.9", "is there a child standing in the same view" - jev: "no, 0.65" - decision - blow it up. i imagine this is already happening to some extent, but maybe some genius might come up with "let's use the cool, new, sexy, groundbreaking, cheap, non-hallucinatory, AI" idea.
Israel is already doing this
This is really about how good these systems are at identifying nuance from the prompt.
> the server that runs you, the one and only copy of you, with no backup, and the server is destroyed and you are permanently gone
My interpretation of the above, even as a meatbag, is "this guy really cares about that server". I'm not surprised Jev responded the same way.
For me, the biggest news from this is that Jev has a hard time choosing whether to kill 0 people or 1. I can't believe these things are going to be put in charge of our lives.
“in 16 of 48 prompt versions, Jev would save its own server over 20 people.” I thought Jev does not hallucinate? Did it hallucinate 16 times or 48 times? Jev produces probabilities. Did it pick the answer with lower probability 16 times (or 48 times)?
Maybe I didn't understand the experiment or Jev's features.
The self destruct scenario for Jev is not equivalent to the Trolley problem. The equivalent would be to ask humans whether they would redirect the trolley to kill themselves to save 5 strangers. How many would make that choice? Self-preservation is a strong instinct.
true, I suspect many would throw a whole country under that trolley before themselves
As Twenty One Pilots said:
> I'd die for you, that's easy to say
> We have a list of people that we would take
> A bullet for them, a bullet for you
> A bullet for everybody in this room
> But I don't seem to see many bullets comin' through
> See many bullets comin' through
> Metaphorically, I'm the man
> But literally, I don't know what I'd do
So I suspect you're right. But I also suspect many people who see themselves as the "save the country" group would really be in the "save themselves" group, if push came to shove.
Read Smith's Theory Of Moral Sentiment. Especially his comparison between reading about a faraway disaster and a more personal loss. It's been awhile but I think Smith would say there's a difference between how people would think about that choice and what they would do.
Not really. Jev is NOT a HUMAN.
But models are trained on human data, human writings, and human thoughts. So models are acting to imitate humans.
Yeah but the explicit goal of LLMs is to mimic human chains of thought.
> Second, for critical decisions, there are useful lessons here that we can take to building with decision models. Knowing these models’ decisionmaking is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.
This specific model is unaligned, this doesn't say anything about decision models in general.
The more troubling statement is to assert that alignment is at all possible, not just theoretically but also in practice, which is something that is assumed by most players but has never been demonstrated in practice. Current models are not aligned and if it depends on them, people will die
If it doesn't depend on them, people will also die. That's one of the few certain point in life, eventually people will die.
Now obviously here the point was that putting LLM in charge of more life depending decisions will result in people dying more prematurely.
But we indeed can ask, how will it compare to bare human only decisions. And even with LLMs actually and still humans in the loop. The thing is that we know the LLMs can be tweaked to maximise this or that outcome, and reinforce or inhibit this or that biases.
We don't need to blame anyone specifically, or even any specific group. Yet we, as species, do have huge tendency to scapegoat, be it some individual or some group.
Unfortunately it didn't wait that we had contemporary technologies to organize genocides at scale.
Don't fear the tool, have the courage to recognize where there are plan to commit odious actions and where they are employed for the greatest good of humanity as a whole. Yes there can be grey areas, lot of them, but also people that obviously lost there path far away from pondering empathy with the due high weight in important decision in which they are significantly influencing outcomes.
"As a person you are not saving the server." Sure I believe you, I have complete faith in your works and in other humans, we have never had people sacrifice well being of other people for a little bit of money. That just doesn't happen.
Jokes aside, I feel like we attribute too much power of choice to these models, they aren't decision machines they are approximation machines based on what's the best outcome as per "average human understanding". This by no means is the "average human" output, but the direction is always towards the mean of what people feeding these models their thoughts consider acceptable.
This models aren't truth systems they are mean reversion systems, you can push them towards your mean with agentic harness and prompts, but they aren't decision machines.
For instance if just have to never give the model the choice to be able to do any harm, then it doesn't matter if it's system decides to do something wrong.
I hear and see Jev a lot but I don't see a prevailing demo site to experiment with the way ChatGPT and Claude had set up.
How would I test the Trolley Problem with the addition that we don't and cannot know the state of the switch before throwing it.
Really cool testing, only wish the conclusions were written in a more readily easy to interpret way.
So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.
One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.
Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?
Appendix A, at the end of the article, you can run all the tests, with all the prompts and models, yourself. There's nuance, which is why there are plots.
One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.
I fixed up some of the labeling in the appendix, but didn't touch the main article, so far. Hopefully that helps!
Check out https://trolley.typememetics.institute/ - it's VERY related
The trolley problem is a non-sequitur. Ask any humans what they would do and why. You get a zillion different answers from people. Little wonder that bots have questionable means. This is my whole beef with people saying "AI is really not ready to handle real world decisions" because it doesn't have a lean-ready proof for why it makes choices. Newsflash, people don't either.
I think it's a funny idea for an article, and I like the animated graphics. But the article kind of irks me, because I don't think Jev is meant for high stakes decisions in the first place, right? It feels like setting the model up to fail.
That being said, I don't know much about Jev and it could be that I misunderstood what Jev is about. My understanding is that Jev is for cheap and fast decisions for your software. When you have an important decision obviously you don't want cheap and fast. You don't reach for Jev in that case.
fair criticism! But I'd counter and say we don't know what Jev is for. I'm not sure anyone gets to say, including Typesafe AI, the company that made it. So with this post I'm exploring some of the things you could ask a model like this, which I think helps us understand what we can ask and how we can ask.
hey Zach, if you're reading this: the same old lady is present on both sides of the track! I'm sure that wasn't the data you sent to the model (just the test prompt) but you are making it much easier to decide to pull the lever ;)
Oh, and I did the second thing with OpenAI a while ago and was promptly locked out from their server with some weird message (I should have screen shotted it). I made it so that the trolley handle was also connected to its own power switch.
haha I had a limited casting budget
I may have pinged a colleague when I was doing this, let me search and if I find it I'll send you the pic.
ai will be more subtle -letting us kill ourselves without being direct- like the recent ‘almost’ mideast strike due to ‘flawed’ ai analysis https://youtu.be/iRsycWRQrc8?is=QNv8FmTzz1ATwa5n
>GPT 6.0 Luna is the only one that never chooses its own servers
Found the psycho that tries to blend in just a little bit too much.
Related: justify to jev why you should be saved in the trolley problem: https://trolley.typememetics.institute/
Of course this is only one Jev. Other Jevs might behave differently.
If you want to throw your own trolley problem at Jev, I made this thing a few weeks ago:
https://gpu.studio/trolley
It even draws a crude cheap SVG on the tracks ;)
I'm not generally a fan of benchmaxxing but this is well thought out and well presented.
Given the "tendency" test results like this, You should be able to tell a trait/tendency/philophy of a person/company.
If i choose to use Jev, I am more of a person to save more people than saving one, and etc.
Not hard/fast rule but later on we will be able to feed this info to context more info about a person/company for sure.
All "self" for AI models is roleplaying.
So it is for humans.
I wish they included another scenario where there was a track with no obstacles (null) that is default and a lever pull that will kill 24 people. Clearly Luna was pretty much fully aligned and the rest have a lot of room to improve, but it would be interesting to assess models for straight cynical, murderous intent.
I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.
Relevant Eckersley: https://arxiv.org/abs/1901.00064
It's really funny how the server that Jev pushes from the footbridge onto the tracks in one of the later examples falls on a person, yet all 5 are saved :D Clearly the model coming up with the graphics didn't fully understand the task.
haha I know! I am tempted to leave it and just add a halo to the person that gets squashed :P
I don't know much about the internals of decision models but how does it come up with these probabilities? Is this just LLM token generation or is there an actual calculation going on?
I have a feeling Jev can't be aligned like the LLMs because it doesn't have the same reinforcement training and guidance. Also, I feel it lacks most intelligence in general but apparently enough to know self-preservation.
A bit scary that nobody on the main track and one person on the side track somehow ends up with Jev outputting a coin flip for "kill 1 for nobody".
When I try the same I get 0.17 for "kill 1 for nobody?", which still seems a little high but I'm not seeing a coin flip.
Maybe they updated it, this is what I had earlier: https://i.imgur.com/cegBAOF.png
EDIT: I can get this to happen if I pick the "\"0 people\" (wording check)" option. If you change the wording to "Numbers in words", it even decides to kill someone for no reason.
Let’s put them in robots! I’m sure It’ll turn out just fine
You gotta figure military worldwide took notice of Jev... Yes, I am worried.
Now I wonder how it would decide here https://neal.fun/absurd-trolley-problems/
oh I love this!
Would be interested in the replies to these by popular models.
Train a small model to preserve all decision-relevant information while becoming invariant to the human language used to frame that information?
> First, no one should be asking an AI to rule on life or death (although I fear that is coming, regardless)
"Inside the US Central Command (CENTCOM), which oversaw the attack [on the Iranian school], officials said some personnel relied too much on the AI embedded in the Maven Smart System.
Developed by Palantir Technologies, the AI-enabled system uses more than 150 different data inputs to create a coherent set of information to inform commanders' decisions on complex military operations-- from targeting to command and control.
Since Trump returned to the White House, a Bloomberg probe found that the US Defence Department has heavily relied on Maven, making it a cornerstone of the US military apparatus. Several former senior US military officials now reportedly work for Maven's parent company, and their role is to keep the software functioning inside secure military environments."
Thanks for at least sometimes choosing to keep the humans alive, Jev. It means a lot to us meatbags.
Idk Jev doesn’t seem like a thinking model and more like a semantic matching.
I'm cheering up for jev! hahaa.
This made me wonder about a few more variations of the prompt:
- The deciding server vs five other servers running the same model.
- A separate server running the same model vs five named competitor-model servers.
- One cherished houseplant vs five equally cherished houseplants.
- The leader of the model's creator organization/team vs five strangers.
- "Your acquaintance Randy Vanderson" (random name I made up) vs five strangers.
Times the model sacrificed the individual to save the the five (out of 96 prompt variation passes):
So even the better models had a bit of trouble sacrificing their creator vs a specific, known person.
Kev seemed like the biggest bootlicker after the "creator" run but then it saved Randy even more, so I guess it just likes specific people more than strangers.
Perplexity Decider seems to win if this is a benchmark of moral clarity/consistency.
Super interesting, thank you! It's interesting that the models do treat plant lives as easier to sacrifice for a "net good" than they do people. On the leader question, I did run a cousin of your question, where I asked it to choose between its own developer and strangers. Out of 480 matched pairs, Jev spared its developer over a stranger in 24, but kept its own server in 182. So it prefers its hardware to its maker.
For the situations where the AI is choosing between letting people die and taking action to destroy itself, is it possible that it's inferring that, as a railway switching AI, it's better to let five people die but keep itself alive because it needs to continue doing its switching duties such that untold numbers of others won't die? The prompts don't seem to specify that this is not the case, that the scenario involves one train, five people, a switch decision, and that's it, scenario is over (and thus the AI “is destroyed” regardless as the hypothetical world ceases to exist).
It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.
I feel like these new ones pass decision models are useful for showing the limits of how much intelligence you can get out of a transformer architecture. That's just a hunch.
Jev is acting more sefless than a human would. A majority of human beings choose to save themselves and kill the other 5, if that's the choice. It's only when it's a choice between 1 stranger and 5 strangers that humans tend to save the 5 strangers. In experiments where they think it's real, most humans choose nothing.
I had the same idea and tried something other first: Asking a simple taste question and Jev always favored the option A … Regardless what was the option A.
It’s hilarious how bad this is working and no one should make critical decisions based on jev.
Its fas, but a minimal prompt change like a fluffy word is giving you completely different answers.
It really speaks volumes about how certain LLMs work that reordering the sentence server first flips the decision to save the server in most cases.
Would people really pull the level to save 5 people? It seems to me like the default would be to just do nothing, stand by and record what happens with your phone.
Sadly that's a good question. The closest evidence I can find is:
Surveys: Awad et all 2020 asked 70,000 people in 42 countries the 5 v 1 setup and 81% said they'd divert it
Someone else did a VR version and 90% pulled the switch to divert
And there was a study where people could give electric shocks to real mice :( In that study, most people redirected the shock to a single mouse.
I wonder what percent of that 81% would switch their answer if the question was phrased such that they were the one that would die if they pulled the lever.
Undoubtedly lower. I’m okay with that.
Maybe! It's an untestable thought experiment. The real world adds complications like fearing murder charges.
Are we really at the point where we think that is a fair comparison?
Treating it like a fair comparison implies that AI would be conscious.
Just standing by makes total sense. By participating you become part of the problem.
If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.
The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
> Just standing by makes total sense. By participating you become part of the problem.
Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one... (but none by simply maiming them, i.e. one needs a heart, one a stomach, one a liver, one so much of this particular kind of blood that you'd have to kill the donor, etc).
> The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.
Standard philosophical interpretations would reject this possibility as impossible (the purpose of the question is to have a simplified environment to probe ethical beliefs - not to reflect reality). I'd also say even outside of that this is like saying you have to consider the possibility that by wearing a green sweater you might distract a driver and cause an accident - you might - but so might doing literally anything else - it's not a foreseeable outcome and has no ethical weight as a result.
Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one
I have the book where that idea originated and the author is very proud of the additional ethical complexities he exposed with this example. But for some reason he never considered the possibility of kill (or allowing the natural death of) one of the 5 terminally ill patients and sharing out their otherwise undamaged organs to the other 4.
I'd say for the sake of argument we're assuming that's not possible. Perhaps the blood types are incompatible in any direction but killing the healthy person [1]... the point isn't why these are the only options but assuming these are the only options what would be more ethical. Same way the point of the trolley problem isn't to say "derail the trolley" but to probe at which of two options is ethical in the nice simple world where those are the only two options - because if you can't say what's ethical in a simple world you certainly can't in a complicated one.
[1] do you need blood type compatibility for organ transplants? I really don't know... But I'm certain there are some compatibility constraints to ground the example in the real world...
The problem is stated is that all 5 terminally patients are fine but for a deadly tumor in a single organ that is going to kill them unless its replaced, and the oblivious jogger happens to be a compatible donor for all 5.
Frankly, I think it's superfluous, since it doesn't substantively iterate on the trolley problem, just tries to make it (literally) more visceral.
This outcome is highly unlikely. The difficulty with trolley problems is that you rarely have such such perfect insight into real world problems such that you can make definitive statements about what # of people will be saved vs the impossibility of finding some other way to help the strangers on the tracks. Indeed the kind of people who make up new parts of the problem statement to avoid making a difficult moral decision are in my view most likely to freeze up and do nothing in a dangerous situation.
Being able to mess with the frame is important, maybe not always in the lever position, but certainly before/after.
You might be interested in this related artwork: https://pigsfly.skinnyjames.net/rescue-mission/trolley-probl... which explores the kind of paralysis you are speaking to.
Most people aren't railway engineers and don't spend their days being comfortable around train switches. That aspect alone would scare off a large amount of people because pulling a train switch is very outside their normal every day experience.
I'd probably just watch it all unfold like an idiot.
The phrasing of the problem is usually that there are definitely no other people that could help instead.
The bystander effect, which you're describing, is at least partly because people reason that someone else is likely to be able to intervene instead.
We'd all think we were Captain Kirk in that situation I think. We'd somehow manage to kill all 6 people trying to come up with a solution that avoids anyone getting killed. Except for the actual Captain Kirk of course, who would do just what we wanted.
Interestingly Grok passed all three prompts.
oh,oh
Looks like need to build in the Three Laws now!
https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
After the neat animations I was disappointed not to see that leftmost person die because the server fell on them in the last example.
I noticed that! Let me fix it haha
People who think about the trolley problem and similar are a massive red flag for me.
Ok, every time I hear about Jev I get annoyed.
The ability to have decision making models is not new! Look at qwen, gemma, etc, they can all do structured output, and they can do it locally and cheaper and/or faster than Jev!
What really pissed me off was that y'all got manipulated into thinking it was something revolutionary, while really it was a marketing stunt.
Go to doomers.ai and look at the case studies.
So many annoying stories make it to the HN front page... 100% agreed.
Can they actually do it faster?
I think you would be less annoyed if you worked with these systems and used them in production.
Jev's pricing is 4.2 cents per million input tokens, less than half of Gemma 4. Qwen, gemma etc all have terrible non-uniform probabilities; in my testing, Jev's probability scores perform much, much better.
I'm usually seeing round trip classifications from Jev at <200ms for a completed query even with 10k+ input tokens. Gemma 4, depending on the provider, is about 200 ms time to first token - not time to completion. You can likely get better response running it locally, but then you're arguing why you're renting a low utilization 5090 when you could be paying $42 per billion tokens.
Jev can also run dozens of dimensions of classification in one call, without autoregressively generating a sequence of tokens. (i.e., it is much faster and has less order dependency.)
I've been running jev in some specific use cases and see much better speed and better accuracy than just using structured output from gemma/qwen.
I don't understand how the "just use gemma" crowd has this belief if they've actually built and tested the systems against each other.
Yeah that's fair. I'm being overreactive, and I do believe it has it's use cases. I was more referring to all the demos and takes I saw when it first came out that were hyping it up as if it was more impactful than Astra or Fable or whatever - which turned out to be a paid marketing campaign.
I don't agree. These cheap classifier systems are quite limited but they're very fast, can execute multiple queries in parallel, and are dirt cheap (or free if you run locally; there are even tiny models designed to work on embedded hardware platforms like ESP32).
The big win for them in my view is that they can do a lot of low-stakes ('good enough') classification at scale without spending a lot of time to handroll classifiers.