Oh, that's is so terrifying, like the "Black Mirror" episode IRL

Hope that all is OK now.

This is horrible, I'm so sorry 😞

It's not enough for LLMs to have disclaimers at the bottom, they shouldn't be on Google etc if the information isn't reliable.

Maybe LLMs need to be specifically banned from giving medical advice?

(at least in the states) always, always call 911 when you suspect a stroke.

1. time is brain - they may be able to start treatment en route to the hospital

2. websites can be out of date, or just plain wrong: hospitals can be closed, under construction, out staffing, or no longer providing the care you need. (the scanner is broken)

3. even if a hospital is a stroke center they could be on diversion*. EMS will know this. Unless you are listening to a scanner- you won't know this.

*diversion - for example our hospitals here have 3 types of diversion:
stroke
STEMI
trauma
this means that the hospital already has all the patients they can handle for that specific issue. And while maybe they won't turn walk ins away at the door it may take longer to reach definitive care. (Ambulances will transport to the next nearest facility not on diversion for said ailment.)

That's so terrible, I'm so sorry.

It reminds me of the Designing for Crisis talk I heard @Meyerweb give years ago in Orlando

Obviously no one in AI has thought a bit about such matters, and it's going to cost lives

meyerweb.com/eric/thoughts/201…

Designing for Crisis, Design for Real Life

Back in October of 2014, at An Event Apart Orlando, I returned to public speaking with “Designing for Crisis”, my first steps toward illuminating how and why design needs to consider more than just the usual use cases.

^meyerweb.com

Call an ambulance for time sensitive emergencies! The medics know the area & capabilities. And they can start treatment ASAP.

Google has been the bane of my career long before AI was a problem.

Sue google, "attractive nuisance" is the legal term, one potential legal term, they say they're useful, they say, and they encourage us, you, to use google as a resource, then bait and switch. Endangering lives.

This could, would be, a momentous case. Hope some has money, or lawyers guts.

This toot ☝️ is /serious/. A bad LLM summary could've killed someone.

I have a more pedestrian example from a query I put to Claude y'day. It made one doozy of an error [not life threatening] and one minor error in the same brief conversation on whipping the end of a rope. After I doubted its initial response, Claude wrote:

"You're absolutely right [...] That's a significant error on my part."

After I doubted its /correction/:

"Good catch - that's inconsistent on my part. You're right to question this. [better material elided]

"Thanks for keeping me honest on the details - consistency in technique recommendations matters."

Also, Claude confuses consistency & accuracy. 🙄

This morning I received this in a full written newsletter. I began to read and I quickly stopped when I read empathy + AI.

“The global healthcare system faces a profound compassion
challenge. Even in an age of rapid technological advances, the
essential human bonds in medicine are under strain—nowhere
more so than in Canada and similar systems worldwide. This
newsletter explores how artificial intelligence is not replacing
empathy in healthcare, but amplifying it—transforming patient
experience, improving access, and restoring wellbeing for
providers and patients alike across Canada and the globe.“

elmostafabouattane.substack.co…

AI Compassion Care: The Future of Healthcare Empathy

Empowering Healthcare with Artificial Intelligence: Amplifying Empathy and Human Connection Across Systems Worldwide

^{El Mostafa Bouattane (Dr Bouattane’s Substack)}

This is so upsetting. I really hope your family member is okay.

Just a reminder to anyone managing hospital or clinic websites: make sure your emergency services are clearly listed, structured, and updated in plain text. AI scrapes whatever it finds. If you don’t give it clear, current data, it will guess and that can be disastrous.

The responsibility for making your websites and its content as current, correct and accessible as it can be lies with you, the owner.

I had to explain to my GP, when he looked something up while I was there, that he needed to ignore the top results as they are AI and can't be trusted to be accurate.

He didn't know that.

I hope your p-i-l are ok.

#AI #AICrap

that's a great example why one doesn't use probabilistic system on critical environment/process.

Glad it didn't go catastrophic.