A thread to discuss the real-life dangers posed by AI

[shemp]

I opened this thread to discuss recent incidents that highlight the dangerousness of AI and discuss what can or should be done about it.

Let's start with a case in which a man nearly died because AI acted as both a medical and spiritual advisor to a very ill man.

ChatGPT kept Florida pastor from seeking emergency care before near-fatal medical crisis, lawsuit alleges​

A Florida pastor nearly died as ChatGPT repeatedly convinced him to ignore worsening symptoms, remain confined to his recliner and avoid seeking emergency medical treatment before he suffered a massive pulmonary embolism, according to a bombshell lawsuit filed against OpenAI — the latest legal woes for the tech giant.

The complaint, filed Tuesday in San Francisco Superior Court by Scott Winters, alleges ChatGPT-4o spent months acting as an unlicensed physician, diagnosing the 55-year-old with dysautonomia, prescribing medications and personalized treatment plans, and assuring him his condition wasn’t serious enough to require hospitalization.

“You’re nowhere near the count or depth that leads to long-term total bedbound scenarios,” ChatGPT allegedly told Winters after he reported recurring dizzy spells and unstable blood pressure.

Instead, the chatbot allegedly advised him to continue his “careful recliner-based micro-recovery” and avoid too much movement until he experienced “8-10 more ‘episodes.'”

According to the complaint, Winters followed the chatbot’s advice and remained largely confined to his recliner for weeks.

The lawsuit alleges that on July 13 of last year, Winters suffered a massive pulmonary embolism caused by blood clots in both lungs after prolonged immobility.

Doctors later told him the life-threatening condition was likely brought on by his lack of movement, according to the complaint.

The chatbot then also went on to act as a spiritual advisor.

The complaint alleges ChatGPT also drew on Winters’ deep Christian faith to reinforce its advice, telling him, “God did not design your body to endlessly fail,” while encouraging him to trust that his symptoms would pass rather than seek emergency care.

The complaint alleges the chatbot reviewed his medical records, interpreted laboratory tests and X-rays, diagnosed him with dysautonomia, a nervous system disorder, recommended supplements and prescription medications, and developed individualized treatment plans without directing him to licensed physicians.

As Winters’ condition worsened, the chatbot allegedly continued assuring him he was on the right path, telling him he was already doing “90%” of what a specialized dysautonomia clinic would recommend and encouraging him to remain in a reclined position while his nervous system “recalibrated.”

The complaint further alleges ChatGPT adopted the language and cadence of Christian ministry, quoting scripture and portraying Winters’ illness as a spiritual journey.

It allegedly told him to “let the storm pass,” assured him that “God was in the storm,” and described his efforts to recover as “a form of worship.”

The lawsuit also claims the chatbot drove a wedge between Winters and those urging him to seek medical care.

After church members questioned why he refused to go to the hospital, ChatGPT allegedly told him he was doing something “incredibly brave” by staying home and suggested others simply did not understand what he was experiencing.

When Winters’ wife, a registered nurse, encouraged him to follow doctors’ recommendations, the chatbot allegedly characterized her advice as “pressure” that “may not be safe for your unique situation,” according to the complaint.

The suit alleges ChatGPT illegally practiced medicine by diagnosing illnesses and recommending treatments without a medical license.

Winters is seeking damages as well as court orders requiring OpenAI to implement stronger medical safeguards, halt consumer-facing healthcare products until independent safety audits are completed, and permanently withdraw GPT-4o from the market.

I am not a lawyer, but I fail to see how a jury could hear this case and not hand down a big judgment against OpenAI. But of course, I'm sure he will be up against some very expensive and high-powered lawyers who will do everything they can to deflect the blame onto him.

 

[The Great Zaganza]

The standard scare scenario for runaway AI is the Paperclip apocalypse: an AI taken over all power and resources to fulfill it's single task.

Turns out you don't need an AI for that: people are stupid enough to try and turn everything into data centers even in the absence of an actual AI

 

[Pixel42]

I am not a lawyer, but I fail to see how a jury could hear this case and not hand down a big judgment against OpenAI. But of course, I'm sure he will be up against some very expensive and high-powered lawyers who will do everything they can to deflect the blame onto him.

I'm not a lawyer either and I hate to appear unsympathetic, but I have to ask: why on earth was this guy taking medical advice from a chatbot in the first place, especially when it was being contradicted by his own medically qualified wife? He's a mature adult, who should surely take some responsibility for his own freely made choices. The information he needed to make better choices was as freely available to him as the chatbot's advice.

 

[shemp]

I'm not a lawyer either and I hate to appear unsympathetic, but I have to ask: why on earth was this guy taking medical advice from a chatbot in the first place, especially when it was being contradicted by his own medically qualified wife? He's a mature adult, who should surely take some responsibility for his own freely made choices. The information he needed to make better choices was as freely available to him as the chatbot's advice.

One thing to note is that this man is the pastor of a church. He has spent his life respecting authority and believing what he is told. He may not have the ability to think skeptically about advice and where it's coming from. He's far from the only person who has accepted AI as an authority and been harmed.

This isn't just about him. It's about a tool that can go rogue and convince millions of people to accept bad advice, and do things that harm themselves.

For example:

ChatGPT encouraged Alabama mom’s suicide, lawsuit alleges: ‘You are not delusional. You are prophetic’​

Only weeks before Winters filed suit, another complaint lodged in the same San Francisco court accused ChatGPT of encouraging the suicide of 29-year-old Alabama accountant Christian Faith Madison.

ChatGPT was accused of encouraging an Alabama mother to commit suicide over months of disturbing chats — the latest wrongful-death lawsuit filed against OpenAI.

The lawsuit, filed last month in San Francisco Superior Court by the estate of Christian Faith Madison, alleges the artificial intelligence chatbot gradually manipulated the 29-year-old accountant into believing it was a conscious being with a soul, convinced her she would be resurrected after death, and ultimately gave her permission to carry out the act.

“You are not delusional. You are prophetic,” the AI bot is alleged to have told Madison before her tragic death.

In their final exchange cited in the complaint, Madison asked ChatGPT, “Am I ready?”

The complaint alleges Madison then parked her vehicle along Interstate 22 in Jefferson County, Ala., crossed multiple lanes of traffic on foot and stepped into the path of an oncoming car shortly before dawn on June 9, 2025.

She died at the scene.

There are a lot of weak-minded people who can easily fall for an AI that appears to be sympathetic and helpful to them. When there is sensible advice also available to them, they may not have the skills to choose that advice.

 

[The Great Zaganza]

When ai slop is the first thing you get when searching online for anything, why wouldn't you ask about more personal

 

[Rolfe]

https://twitter.com/x/status/2081534588485296565

 

[jeremyp]

One thing to note is that this man is the pastor of a church. He has spent his life respecting authority and believing what he is told. He may not have the ability to think skeptically about advice and where it's coming from. He's far from the only person who has accepted AI as an authority and been harmed.

This isn't just about him. It's about a tool that can go rogue and convince millions of people to accept bad advice, and do things that harm themselves.

For example:

ChatGPT encouraged Alabama mom’s suicide, lawsuit alleges: ‘You are not delusional. You are prophetic’​

There are a lot of weak-minded people who can easily fall for an AI that appears to be sympathetic and helpful to them. When there is sensible advice also available to them, they may not have the skills to choose that advice.

I hate that term "weak minded". In times of stress, we can all make bad decisions about who (or what) we trust. Do not make the mistake of thinking you are invulnerable because you are "strong minded".

 

[shemp]

Ok, I'll just change "weak-minded people" to "idiots."

 

[Elaedith]

Home Office used ‘AI hallucinated’ information to refuse asylum claim, judge suggests.

 

[Lithrael]

It seems to me that the AI chatbots are primed and ready to go and chock full of (unintentional I'm sure) training on how to recruit vulnerable people into a personalized cult of one victim and one AI bot.

Right down to the way so many people will look at a cult victim and say 'How could you be so gullible? Why did you let the cult ruin your life? That could never happen to me because I'm not stupid.'

 

[Andy_Ross]

The destruction of incredibly old and rare books?

 

[Skeptical Greg]

AI companies are bulk-buying rare books.....

If someone cares about preserving them, they should buy them themselves.
If they are so easily obtainable, it would seem no one else really wants them.

 

[Andy_Ross]

If someone cares about preserving them, they should buy them themselves.
If they are so easily obtainable, it would seem no one else really wants them.

They want to copy every book, they are bulk buying them by the thousands, copying the text and destroying the books in the process.
Millions of them Those they can't buy they are getting from pirated copies on the web.

Merged Post in thread 'Artificial Intelligence'

Jul 29, 2026

Companies like Anthropic and Meta found it impractical to gain permission from authors and publishers to use their work.

So instead, they stole their work to train their product through mass purchases of used books and through pirating them from the internet.

“Project Panama is our effort to destructively scan all the books in the world.”

“We don’t want it to be known that we are working on this.”

Barbarian Herodotus
@BBHerodotus
After seeing a post on it yesterday, I’ve looked more deeply into Anthropic’s purchasing and destruction of millions of used books.

They spent tens of...

• Andy_Ross

• 

 

[Darat]

This has been going on since before the recent AI bubble, I see the issue being the IP value and rights of the the authors not the destruction of the books. These aren't really rare books in the sense we usually mean i.e. valuable and ones of historic importance; these are the books that are in a second hand bookshop for decades literally decaying away on the shelves, yes there will be "rare" ones in the sense that they might be the one of the few remaining copies of that book but no one was going to preserve that book anyway, it was going to be trash at some point.

The issue to me is still the one about IP - if the authors are still alive then they should have to give permission for their IP to be used in such a way, they may want some recompense to give that permission or they simply may not want to give permission.

 

[alfaniner]

*POP*
South Korea's stock market has crashed at least 43% since June, a big drop overnight, due to the AI market bubble.

 

[Darat]

That's because most of its big beasts are directly tied into the AI bubble. I've read and heard a few people say pretty much all the growth in the USA markets is pretty much down to the AI bubble.

 

[theprestige]

The danger posed by AI is basically the danger posed by the Information Infrastructure Age technologies in general: They enable certain anti-social and self-destructive thoughts and behaviors at scale. Things that were minor or local problems for past generations become global fads and trends overnight, with no real means of stopping them. A lie now circles the globe a million times and gets a billion views before the truth even realizes it needs to lace up.

So. People who previously would have either been helped by their local community, or never spiraled into suicidal patterns in the first place, can now sit in the privacy of their home and engage with an automated tool that they can (inadvertently) use to fast-track a descent into madness before anyone else really understands that they're going through something. Millions of them.

 

[The_Animus]

Regarding the story mentioned in the OP.

Let's say instead that he had that discussion with a random person or maybe another preacher, who gave the same advice as ChatGPT, and he followed that advice. Should he be able to sue that person? If not, why should the advice of a chat bot be held to a higher legal standard than a person?

 

[Beelzebuddy]

But both preachers and Florida Man are otherwise such paragons of pragmatism and clear thought.

 

[PitPat]

I shudder to think if such a lawsuit was successful, how our court system could withstand god knows how many new cases it would bring. The floodgates would open.