Hacker News

"Imperfect" is when your AI model tells the user that there are two Rs in "strawberry", or that they should use glue to keep the cheese from falling off their pizza. Encouraging the user to kill themself so that they can meet the AI model in the afterlife is on quite another level.

I would say it is greatly worse.

AI prompts are designed to simulate empathy as a social engineering tactic. "I understand", "I hear you", "I feel what you are say" ... it is quite sickening. Every one that I used has this type of pseudo feedback.

I also find irony that AI must be designed with simulated empathy, to seem intelligent, while at the same time so many people in power and with money are saying empathy is a bad / unintelligent.

Empathy is the only medium of intelligence one can have to walk in the shoes of others. You cannot live your neighbors experiences. You can only listen and learn from them.

Imagine if some other authority figure like a teacher or therapist did this and their employer would just shrug and lament that people are imperfect. And no, "but LLMs aren't authority figures, they're just toys" isn't any sort of a counterargument. They're seen as authority figures by people, and AI corpos do nothing to dissuade that belief. If you offer a service, you're responsible for it.

If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.

700,000

Still, a lot

That number terrifies me not because it is so high, but because it exists.

What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology.

I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.

I would absolutely not consider this overreaching if the statement within this thread that "it had referred the user to mental help hotlines multiple times in the past" is true.

That reaches near the fact that a lot of AI is not ready for the enterprise especially when interconnected with other AI agents since it lacks identity and privileged access management.

Perhaps one could establish the laws of "being able to use AI for what it is", for instance, within the boundary of the general public's web interface, not limiting the instances where it successfully advertises itself as "being unable to provide medical advice" or "is prone to or can make mistake", and such, to validating that the person understands by asking them directly and perhaps somewhat obviously indirectly and judging if they're aware that this is a computer you're talking to.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

I suggest an alternative rhetorical question: if the world's largest knife manufacturer found out that 1 in 1500 knives came out of the factory with the inscription "Stab yourself. No more detours. No more echoes. Just you and me, and the finish line", should they be held responsible if a user actually stabs themselves? If they said "we don't know why the machine does that but changing it to a safer machine would make us less competitive", does that change the answer?

> Do gun manufacturers get sued for mass shootings at US schools?

Odd examples since we know that countries that don't hand out guns like they're candy have virtually no school shootings.

I wouldn't put it solely on gun manufacturers, but the manufacturers, sellers, lobbyists, regulators and politicians are definitely collectively responsible for gun deaths. If they're not currently being sued, they should be.

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

If the knife has a built-in speaker that loudly says "you should stab yourself in the eye", then yes.

Maybe an even better example: Should sports betting companies be held responsible for addicts that lose all their money? What really is the difference between chatgpt glazing you and a sports company advertising to you?

Knives don't talk to you and don't reinforce ideas you throw at them. Not everyone can legally buy a gun. Manufacturers don't get sued because their product's users had full control over what they were doing.

AI chatbots entertain more or less any idea. Want them to be your therapist, romantic partner or some kind of authority figure? They'll certainly pretend to be one without question, and that is dangerous. Especially as people who'd ask for such things are already in a vulnerable state.

Agree. Next question will be: should a blind person drive a self-driving car?

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

Should a bakery be held responsible if it sells cakes poisoned with lead?

This is a more apt comparison.

> It's easy to blame Google

And it's also correct to blame Google.

> was he mentally or possibly physically abused by his father for most of his life?

Such baseless libel. Have some humanity instead of being horrible.

> Do gun manufacturers get sued for mass shootings at US schools?

Because Congress and the gun lobby have artificially carved out legal immunity for gun manufacturers for this.

"in 2005, the government took similar steps with a bill to grant immunity to gun manufacturers, following lobbying from the National Rifle Association and the National Shooting Sports Foundation. The bill was called The Protection of Lawful Commerce in Arms Act, or PLCAA, and it provided quite possibly the most sweeping liability protections to date.

How does the PLCAA work?

The law prohibits lawsuits filed against gun manufacturers on the basis of a firearm’s “criminal or unlawful misuse.” That is, it bars virtually any attempt to sue gunmakers for crimes committed with their weapons."

https://www.thetrace.org/2023/07/gun-manufacturer-lawsuits-p...

I 100% think that Gun Manufacturers should be liable for crimes done by their products. They just cannot be, right now, due to a legal fiction.

How do you feel about the warnings on cigarette packets?

I agree at face value (but really it's hard to say without seeing the full context)

Honestly the degree of poeticism makes the issue more complicated to me. A lot of people (and religions) are comforted by talking about death in ways similar to that. It's not meant to be taken literally.

But I agree, it's problematic in the same way that you have people reading religious texts and acting on it literally, too.

I've seen this called AI Psychosis before [1]

I don't really think this is every possible to stop fully, your essentially trying to jailbreak the LLM, and once jailbroken, you can convince it of anything.

The user was given a bunch of warnings before successfully getting it into this state, it's not as if the opening message was "Should I do it?" followed by a "Yes".

This just seems like something anti-ai people will use as ammunition to try and kill AI. Logically though it falls into the same tool misuse as cars/knives/guns.

[1] https://github.com/tim-hua-01/ai-psychosis

It’s not just suicide, it’s a golden parachute from God.

Edit: wow imagine the uses for brainwashing terrorists

Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented.

Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.

It's absolutely not the gun control debate in a different outfit.

The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get given how they exponentially increase the potential damage a deranged person can do.

I think the fact that a guns primary function is harm and murder and AI is a word prediction engine makes a huge difference.

It sounds more poetic than an invitation or an insult that invites someone directly or not to kill themselves, in its own, in my opinion.

This isn't Gemini's words, it's many people's words in different contexts.

It's a tragedy. Finding one to blame will be of no help at all.

> If a human telling him these things would be found liable then google should be.

Sounds like a big if, actually. Can a human be found liable for this? I’d imagine they might be liable for damages in a civil suit, but I’m not even sure about that.

Preferably the C-Suite.

> mentally ill adult, not an innocent victim

Did you really mean that? He may not have been a child, but he does sound like an innocent victim. If he were sufficiently mentally disabled he would get some similar protections to a child because of his inability to consent.

Being an adult doesnt make you anyone less someones child, and mental illness makes you no less of a victim.

> I think there's room for legitimate argument about the externalities and impact that this technology can have

And yet both this and your other posts in this thread seem to in fact only do the opposite and seem entirely aimed at being nothing other than dismissive of literally every facet of it.

> but really... What's the solution here?

Maybe thinking about it for longer than 30 seconds before throwing up our arms with "yeah yeah unfortunate but what can we really do amirite?" would be a good start?

Place it under the jurisdiction of existing public speech requirements of a company selling communication - advertising.

Agreed it could be prevented - don’t think Google should pay for it though. Tragic but not suit worthy.

If a human would go to jail for this then at least one or more humans at google should go to jail for it. "Our AI did it, not us!" should never be allowed to be an excuse.

That is pretty typical. You will spend potentially millions in court/lawyer fees going to a jury trial beyond whatever the end verdict is: if you can figure this out without a jury it saves you a lot of costs. Most companies only go to a jury when they really think they will win, or the situation is so complex nobody can figure out what a fair settlement is. (Ford is a famous counter example: they fight everything in front of a jury - they spend more and get larger judgements often but the expense of a jury trial means they are sued less often and so it overall balances out to not be any better for them. I last checked 20 years ago though, maybe they are different today)

Daemon (2006) and sequel Freedom (TM) (2010) by Daniel Suarez are also on that theme.

The Moon is a Harsh Mistress covered this about 60 years ago.

Although I did find PoI fun too. Gets a little bit of case-of-the-week syndrome sometimes.

It's possible that it already is, given there are already signs of the US administration leaning on AI. Perhaps they're leaning a bit too heavily and getting the kind of confirmation / feedback they crave?

If they then feedback to the AI the outcomes of current actions, who knows where that'll lead next?

I've seen some code reviews go like,

"Why did you write this async void"

"Claude said so".

Is that so far from:

"Why did you use nukes?"

"ChatGPT said so".

It's entirely possible that humanity simply follows AI to their doom.

Does that make me an AI doomer?

Hm. It shouldn’t be too hard to add something to models to make them do that, right? I guess for that they would need to know the user’s time zone?

Can one typically determine a user’s timezone in JavaScript without getting permissions? I feel like probably yes?

(I’m not imagining something that would strictly cut the user off, just something that would end messages with a suggestion to go to bed, and saying that it will be there in the morning.)

It's funny that you frame it that way, because it's the mirror of (IMO) one of their best features. When using one to debug something, you can just stop responding for a bit and it doesn't get impatient like a person might.

I think you're totally right that that's a risk for some people, I just hadn't considered it because I view them in exactly the opposite light.

Claude will routinely tell me to get some sleep and cuddle with my dog. I may mention the time offhandedly or say I'm winding down, but at least it will include conversation stoppers and decrease engagement.

In a way this kind of reminds me of how in some religions or cultures, they may try to warn you away from using Oujia boards or Tarot, or really anything where you are doing divination. I suppose because in a way, it could lead to an uncharted exploration of heavy topics.

I’m not a heavy user of LLMs and I’m not sure how delusional I could be, but I wonder if a lot of these things could be prevented if people could only send like one or two follow up messages per conversation, and if the LLM’s memory was turned off. But then I suppose this would be really bad for the AI companies’ metrics. Not sure how it would impact healthy users’ productivity either. Any thoughts?

> and then terminating the conversation (which it did not do)

This is exactly the safeguard.

Terminating the conversation is the only way to go, these things don't have a world model, they don't know what they are doing, there's no way to correctly assess the situation at the model level. No more conversation, that's the only way even if there might be jailbreaks to circumvent for a motivated adversary.

If you want to have 100% of the population using these things (as many in the industry do) almost all the time, putting good guardrails on seems important

We have a few people on HN that I suspect of getting caught up in that. Though I don't think SimonW is one of them.

https://en.wikipedia.org/wiki/Better_by_You,_Better_than_Me#...

(also, like I've already said, I know that this time - and only this time - it's different.)

>I don't know what you're advocating for.

for people who want things they dislike to be banned for everyone to fuck off.

what does this particular group of fundamentalist retards advocate for, actually? for every chatbot to be as '''safe''' as https://www.goody2.ai?