Hacker News

There are 3 kinds of mechanics:

Scammers who do the lowest effort diagnostic and "fix" to get you to pay a smaller amount of money to fix the problem in the short term even though it'll re-present itself a week/month/year later.

Upsellers who will find other things "wrong" with your car and pressure you into paying to fix them because they sound a lot worse than they are.

Good mechanics that will explain what they did to diagnose the issue and recommend different options depending on what the issue is.

Funnily enough, I've found that doctors tend to also fit into these 3 archetypes.

I have multiple LLM subscriptions at any given time, plus an array of local models.

When I ask a question outside of my domain of expertise I like to ask all of the LLMs I have access to. I also create separate sessions and ask the same question multiple ways.

It’s revealing to see how many different and contradictory answers I get, most of which are presented confidently.

The last time I ran a medical question through Claude I couldn’t even get consistent answers between sessions.

It’s also scary how easily you can lead each LLM to the answer you have in mind. When I would start asking questions about different options that other LLMs had presented, each session would drift toward that explanation.

The best mechanic I ever had kept my ‘98 Subaru going past 200k miles. Once during a repair I asked him to do an inspection and tell me if there was anything else I should replace. He told me not to do that, and that any mechanic would always find something, but not necessarily the next thing to break.

He said it better using an expression I hadn’t heard before or since, something like “don’t go looking for goats when your herd is already with you.”

There's a big difference between a _puzzle_ and a _mystery_. In a puzzle, the goal state is known, and as more pieces - data - appears, the goal gets closer. You know how far you are from the goal.

A mystery is worse. With each additional piece of data, the goal gets farther away. Everything is more and more confusing.

(Popularized by Malcom Gladwell)

> The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.

I'd argue that AI _can_ currently provide that, but that it can't do it _reliably_, and that to non-experts it's impossible to differentiate, which makes it all the more dangerous.

The soothing sound of ChatGPT telling us how right and clever we are…how could it possibly hallucinate, certainly not 5.5

To provide a competing point of anecdata: A Gemini diagnosis saved me $3,000 in unnecessary repairs on my Civic.

I tried that AI diagnosis for my 15 old Ford C MAx too, however with a diagnostic problem the issue is unless you've got the ground truth, there's simply no way to verify any tool / human with a metric that you can compare and decide on future tasks.

The AI might be very good at diagnosing all minor issues, but might not lead to a successful repair, whereas human mechanics are extremely good on 80% of major issues that's not the ground truth, but will lead to successful repairs (that might not address the root but simply patch it). So it comes down to manage expectation / outcomes.

These tools can’t reliably fix a 4px misalignment on my icon, better ask them about a medical report… but honestly, I would do the same.

You nerd sniped me with the story about your used car. What happened in the end? I really want to know! There are some fun YouTube channels that basically do the same. Someone who is an expert auto mechanic takes a used car to various repair garages and asks them to recommend a course of action.

> I got 3 completely unrelated recommendations, including one that I knew was invalid! I felt worse off than when I started!

I would frame it differently: you now know which shops are not to be trusted. So, next time you need one, you will take a better decision.

> I got 3 completely unrelated recommendations, including one that I knew was invalid! I felt worse off than when I started!

I almost had a very similar experience with my beater Lexus. It took 2 independent shops and 3 dealers to finally figure out what was causing the ABS to go off randomly at low speeds. Turns out there's some obscure Toyota-specific tool from the late '90s that picked up a proprietary diagnostic code, and the third dealer was the only one that still had that particular piece of equipment.

...and of course, the thing that's broken has been out of production for 20 years and remanufactured ones cost more than the car is worth. I ended up just unplugging the ABS control module.

Point being: once I knew what was wrong, all the seemingly contradictory information from the other 4 shops suddenly fit together. It's just such a weird thing to go wrong that no reasonable tech would ever have considered it.

Especially in the medical field where the placebo effect / mindset shapes outcomes.

> The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.

Aside from the LLM-ism (it isn't foo, it's bar) - this is a thought terminating cliche. You definitionally don't know if some information is better or not given that you were uncertain about the information in the first case.

"I went to three mechanics and got three different answers" - your takeaway is just "Ah - I clearly need better informed mechanics."

Which is on it's face absurd because if you could clearly judge the ability of the mechanics you wouldn't need their evaluation. You'd just do the evaluation yourself.

> it's better information, and AI cannot currently provide that

It sometimes can, if it straight out never can no one would use it. People use it , lots of them.

> There's something incredibly peaceful about being in the hands of an expert you trust

This is the primary business model of enterprise IT and is why companies pay so much for 4 hour disk replacement.

You only got 3 opinions on your car? Why not 50? You could have found a more useful signal by getting more information.

I get it - getting an opinion from a mechanic is time consuming. Not true of AI though.

> AI can hallucinate, but I've been misdiagnosed by humans so many times too...

I've heard this experience from quite a few folks before, but this is my first time hearing about a mandatory 8 months quarantine as a consequence... damn

How is it possible? You can't diagnose tuberculosis just based on imaging and tuberculosis hospital has to know that!

I had a similar experience. My son had pneumonia and was still filling pain after 10 days of antibiotics. Took an X-Ray to three different doctors, and only one got the right diagnosis (pleural effusion). It's really something we should have a central place with top notch professionals looking at it, instead having each doctor to find by themselves.

A second opinion is a smart move if one has doubts about their diagnosis. Doctors make mistakes, and even though I've worked with countless great doctors, I've never worked a job where there wasn't at least one who was undiscerning, or downright lazy and negligent. It's hard to tell people to trust their doctor when I know there are plenty of doctors out there like this.

But AI as of right now is worse than any bad doctor I've ever worked with.

What country / municipality are you in? This is not my understanding of Tuberculosis...

Your TB stories made me recall my (fond) TB stories. I came from a country that requires tuberculosis vaccines as a school-entry requirement. I have the vaccine and antibodies. Then moved to a country that didn't have this requirement but I had to be tested to make sure I didn't have TB for things like camps, college, etc. The test is something like a vaccine that injects only dead TB cells, if the injection site welts up from anti-bodies then you had to get another whole panel of tests (like X-ray of lungs). Thankfully I've never had it but TB is apparently no joke. Though blasting my chest with radiation is no healthier :p

Anyway, thanks for sharing!

Asking for a friend, who is in a somewhat similar predicament — it wasn’t Portugal, was it?

Absolutely. It's very unfortunate that this post used the worst example possible of using LLMs for medical purposes.

General-purpose LLMs are _fantastic_ at medical diagnosis that do not involve imaging. I am completely convinced that given enough information and time, frontier models already outperform >90% of doctors on initial diagnosis of internal issues and suggesting medical tests to further reject or confirm the most likely theories. To the point where I'm eagerly waiting for the first hospital in the world that's willing to be open and honest about using them for that first step, and then proceeding from there. I'll be on a flight there as soon as one arrives.

At the same time, they're worse than useless at anything involving medical imaging. Asking them to interpret them is worse than trying to interpret them yourself as a layman. And you surely wouldn't interpret them yourself.

I don't have insider information, but: if one of the AI companies really wants their models to become really good at this and publicly available datasets are scarce, they can probably just buy anonymized X-ray/MRI scans paired with the human doctor's diagnosis, and train on them. I don't know what the legal story is around this, but AI companies have near infinite money, so I'm sure they can buy their way around regulations (eg. by buying them from a less regulated country).

Yeah, medical computer vision is a (fascinating) field with a lot of ongoing research. SOTA models are highly specialized, and are only getting good enough to be used by actual doctors and patients. Using a general purpose LLM to do this is similar to giving a credit card to Openclaw and telling it to make you rich through the stock market & cryptos.

No trolling here: Do you feel threatened by the advance of AI/LLMs with respect to your field? I would. I am a computer programmer, and it absolutely feels threatening.

I can see how your thesis is valid.

Like OP, I also had a shoulder MRI, and asked two AIs for opinion (awaiting a follow up appointment to discuss the results).

They both insinuated much more serious problem than it was (as judged by an orthopaedic doctor).

Anecdotally, I've had Claude (Sonnet and Opus latest) consistently misread numbers from screenshots of my macro tracking app. Makes me skeptical of claims about its usefulness for anything requiring accurate image interpretation, let alone MRI analysis.

> It funny to see the community here expects the human body to be treated like a deterministic function

In a community largely made of people whose job it is to produce such functions, I'd say it's to be expected

I'm not sure what your point is. Are you saying that medicine is inherently fallible and therefore AI is more likely to make a good diagnosis - particularly a cluster of specialist AIs?

> So an ultrasound report can state there are no calcifications while a plain radiograph can report the presence of calcifications without being inconsistent. Obviously very confusing to patients and people unfamiliar with medical jargon

This is being overly nice, I think. Anyone who doesn't understand this is an idiot imo. You would have to assume that every type of diagnosis instrument has infinite clarity and is always correct to be confused in this case.

Reminds me of the Babbage quote where somebody asked him, if I put the wrong question into this computing device, will it still give me the right answer? His response, paraphrased "I can not fathom the logic of the minds which would come up with such a question".

As a rad tech, YOU TELL ‘EM DOC! I do like some uses of AI I’ve seen that help patients advocate for themselves or understand basic things like blood panel numbers, but it’s really bad at glazing people and leading them down medical rabbit holes kind of like the OP.

You would think that the AI would point out that calcium is best demonstrated on Radiographs/CT imaging vs Ultrasound or something to that effect.

Agreed. Not a radiologist, but I do a fair bit of MRI research. Experts vs lay people probably have different success with getting the right diangosis out of a frontier model. Subtle changes in prompts can cause different diagnosis[1]

[1] https://www.nature.com/articles/s41591-026-04501-8

Radiologist who does read shoulder MRI would like to add that over half the annotations are wrong, glaring mistakes in anatomy and cardinal direction which begs the question of how is it making these findings without knowing what it’s looking at (here’s a hint, it’s hallucinated based on reports it sees).

Can vouch for it. Ultrasound hasn't found calcification in my shoulder but MRI did. Exactly as you said, because it was very small.

Huh, I'm reading and looking up these words you guys are saying and it is starting to look exactly like the symptoms I have been having with my own right shoulder! I feel like a giant gaping rabbit hole just opened up next to my desk.

Why isn’t diagnostic ultrasound used in orthopedics? They inspect fetus hearts and other organs everyday, why not shoulders? Seems much cheaper and faster.

So Opus might be correct?

> I'm a radiologist

Any comment that doesn't start with this or similar qulaification should be taken with a grain of salt (yes, including this one).

Medical imaging is one of those things everyone thinks is simple because they don't know what they don't know. I'm a cardiac sonographer, and I have to assume radiologists hear at least as many eye-rolling takes on AI coming for their job as I do.

Does radiology really make +$700,000.00 a year ?

Someone on reddit claiming to be a radiologist claimed that.

I wonder where the savings will go when those jobs are gone.

I have a story about this very issue.

I have hip impingement, I played sports and got a labral tear in one of my hips. My hip would get sore and painful after a lot of activity. I've seen a top surgeon in the US. After we just met, he looked at the MRI(yes there was a small labral tear there) and said he can have me on an operating table in 2 weeks.

I was shocked, because the recovery absolutely sucks. So, I got 2nd and 3rd opinion.

3rd opinion was a doc with 20+ years of experience. Asked me if I plan on going pro in any sports, I said no, he said the surgery is not worth it. I did some PT and barely have issues with that hip.

Then, Obama admin created a website to see what gifts($) physicians accept. The 1st surgeon had accepted six figures+ from stryker. The older doc? 0.

There is no money in PT for a surgeon. I would thread lightly with popular and young surgeons.

Personally I've always appreciated talking to nurses I know.

The respectable ones know they aren't doctors, but they've seen a lot more recoveries and cases where minimal intervention was required. As some people have said some surgeons like to cut people up.

I had issues with my shoulder for years. Tried PT as well as pull/push-ups but doing that made the pain worse (if I wasn't doing any exercises involving the shoulder it was "fine")…

On the flip side, when I had rotator cuff issues, the surgeon recommended months of physiotherapy before resorting to the knife. And it worked. And by weight training regularly with a focus on correct shoulder movement, the pain stays away.

It really seems like if you, as a patient, go looking for a quick fix, that’s what you’ll be offered. And if you educate yourself a bit and then go t for the best fix for you, you usually get they.

    > Let me just try taking a second-gen allergy tablet every day and see what happens.

Ok, there's a lot to unpack here and you really had the deck stacked against you. First, lets go from the top, once a test says X, disproving that X is really hard. And that's not unique to the medical profession, it's inherent to all humans and we suck at revisiting or revising our decisions, much less at looking at the possibility to even reverse it.

Which moves us to the next two issues: liability and time. Any moment that you ask someone to revise a decision and specially with the stakes that the medical profession has that nobody has the time nor the inclination to open themselves for a mess.

Now, if you really want to be successful, you have to, before they even have a case with you, and specially before the diagnostic loop closes, to suggest the tests that the study has, since that has the biggest chances of looking at the right thing to look. Just be straight that you walked in with a theory. Doctors notice when they're being steered way faster than they notice when you're actually right. That's how you work with the systems that have a overworked mass trying their best.

Daily allergy tablets are associated with huge increases in early onset Alzheimer’s. Glad you found something that works, but might be good to get some of the allergen injections :)

AI makes up for its poor reporting by enhancing the images.

Current Siemens MR software ‘Deep Resolve’ makes up the signal (adding about 50%), then makes up every second pixel, and then, for 3D sequences, makes up every second slice. It’s locking about 59% of the time off each sequences. And it’s really really good. I’m an MR tech.

It's like people who expect ChatGPT to be really good at chess because chess engines with super-human performance have been around for decades, so obviously the latest frontier LLM that took billions to train should find the task trivial.

Actually, I'm curious what ChatGPT 5.5's ELO is- I wouldn't be too surprised if it's 2000+ just from its basic understanding of chess principles from all the content it has digested.

older vision LLMs chopped up images into patches which were projected into the same embedding token space as words. Newer ones use an encoder to more efficiently project an image into token space. Then it runs through the same attention layers as the text component.

No, it does not work like that, it actually can process the image itself there is not an intermediate image to text step

> I do think that people saying “doctors don’t know the state of the art” have a weaker case.

This is kinda the case though. In Poland I met only one psychiatrist that knew about DSM-5. In this year. DSM-5 was a thing from 2013.

Doctors are people just as us, not every single of them is good.

Well I live in the nightmare that is the Dutch healthcare system [1]. There are many things that they will fix but they didn’t fix my sleep. A friend fixed my sleep. He is a doctor and prescribed me the right thing. The thing is, he shouldn’t have had to intervene. Without him I could have ended up poor and destitute as my sleep was wrecking me.

And yea, I already did all the standard things. CBT for insomnia helped somewhat. My insurance didn’t fully cover it either, unless I was willing to wait for 8 to 12 months.

And I recently met someone with slow moving metastatic cancer. Thanks to LLMs they will most likely live another 3 to 5 years extra since the Dutch conventional mainline treatment hasn’t been taken yet. But it is German doctors that helped them and Belgian doctors that pointed out in a second opinion that a lot more can be done.

LLMs have a part to play. The false positives are awful, but I have seen an average of 5 out of 10 care when things become too complicated.

Except for trauma treatment. The Dutch healthcare system is amazing once they diagnose classic PTSD.

So it’s definitely not all bad but the trust I had when I was younger has been eroded quite a bit and LLMs can meaningfully step in, in my case at least.

[1] I know there are worse systems. But from what I have heard there are clearly better systems nowadays. It has slipped a lot

The NYT did this profile a while back: "Ben Riley was already writing about the risks of chatbots when his dad started trusting A.I. over his doctor."

The dad was a retired neuroscientist who delayed cancer treatment against medical advice because he was certain he had been misdiagnosed based on his own research that he did with the help of A.I.

https://www.nytimes.com/2026/04/13/well/ai-chatbots-cancer.h...

There's a comment on the article from Ben Riley:

> I am very grateful to Teddy Rosenbluth for sharing my father's story with the world, her kindness and curiousity proved to be restorative in ways I didn't anticipate.

> The two words that everyone used to describe my dad: "intelligent" and "kind," and he was indeed both of those things. The sad irony here is that it was his human intelligence, combined with these strange new tools that purport to be a form of 'artificial' intelligence, that led to his ill-advised decision to forego the treatment he needed for his CLL. A doctor has already commented on this story with the observation that AI "confidently asserts erroneous conclusions," and we simply have no idea how often this is happening or the magnitude of the harm that results.

> Not a day goes by that I don't feel the pang of my father's absence. He might still be here if not for AI. I try not to think about that, but sometimes I can't help myself.

I asked a clanker about symptoms I was having. (I'm not an idiot, I was already on my way to hospital, clanker was just to take my mind off symptoms during the drive.)

The clanker said I'd be fine, I just needed some rest and OTC meds.

The medical staff immediately turfed me to surgery because the same set of symptoms I told the clanker were enough to concern them that I needed emergency surgery.

Had I have listened to the clanker, I'd be dead because I did need emergency surgery. (Hell, I almost kicked the bucket because I waited for someone to wake up to give me a lift because.my insurance probably doesnt cover an ambulance ride.)

It's not just the second-guessing. It's the getting in the ballpark but striking out: explaining in detail why they are not correct. A little bit of patient knowledge requires a tremendous amount of doctor time to explain away the ignorance.

It's a 180 for me: While I believe doctors should explain diagnosis or treatment decisions when asked, I don't believe they should be taxed with explaining away alternatives. In my anecdotal 2nd- and 3rd-hand experience, doing that is taking at least a third of their time (on roughly 5% of the patients who think demanding answers will make things better) -- with zero improvement to diagnostic accuracy or treatment effectiveness. Doctors already consult with other doctors, and it makes no sense for them to have to consult with ignorant patients or treat their AI psychosis on top of their disease. It doesn't increase patient autonomy any more than adding a steering wheel for child car seats would help toddlers learn to drive.

It’s funny every profession deals with customers making their own guesses at diagnosis.

I told my mechanic the film flam is broken but he said it was the rim ram. He fixed it and we all went in with our lives.

But doctors insist on this God like status so it’s a “nightmare” when patients try to help themselves.

Nightmare because they're always right and the A.I second guessing is always wrong, or because they just don't like to be second guessed?

This is obviously going to happen. But sub-par and sloppy doctors are a thing too. Medicine has been using semi-intelligent systems for years that were nevertheless found to improve outcomes.

We need studies that quantify error rates from each source type, then we need to account for the fact that the artificial type will keep improving.

Indeed. I don’t even get what OP thinks they are getting out of this other than doubt.

It can be helpful in your understanding the choices made by asking questions and thus in reassurance, but it requires something most people lack: understanding you are likely wrong since you are just collecting information without understanding it.

Pretty much the like most manager these days, so I understand the frustration of the GPs.

People should've googled their symptoms and especially the prescriptions they got. It has always been a good practice. If[0] AI proves to be the new google then people should ask AI too.

[0]: IF.

And say it's true because the AI said so.

It's so much worse than some Google results: people see LLMs as a trusted friend who never talks back and never questions you, who is excellent at convincingly communicating their bs, reeling you in with "tell me more so I can really lock this down", continuing to fool you

A con artist, a fraud

No, this flow is actually very good.

Like any domain, when you have questions or need a solution, you make research first, then you ask a specialist.

If you explain well the symptoms and context you can have proper advices and then decide on the path next:

    Case A) It looks benign and advices / information that you collected seem reasonable, then you go your way.

    Case B) You need second opinion of a specialist because the subject is too complex, or there are medications that you need approval.

At home you have the time to assess the options, pros and cons of each approaches, the possible questions to ask and then challenge the doctor.

Shared decision-making is an actual evidence-based model of care, and patients who arrive understanding their condition and carrying specific questions tend to get better attention and better outcomes.

Some doctors get annoyed, because they have big ego and choose to be patronizing, but it is exactly their job to answer such questions.

    With LLMs, it's quite good, you get nuanced and rather useful answers.

    Before LLMs, no matter the topic you searched for, the answer was the same: "you have cancer / an [obviously deadly] rare disease"

    • The doctors are not affordable
    • They are too busy for you (< 15 minutes)
    • You may need to wait months to get an appointment
    • They are not good (country-side is an example, and sometimes even country-level)

So, you have something deeply bothering you, your only appointment is in 4 months. It would be insane not to take the time to explore different solutions and not to come informed about the topic.

If you express your prompt properly and do not rely on imagery, you can absolutely have top-tier advices.

I think that much of the visual gap is because what to attend to in images is less structured. Anecdotally small qwen finetunes (ie less than 10B) take task accuracy from sub 30% on FMs to 90%. We have sold some of these for outcome based back office tasks.

I think we’ll see a lot of specialized VLMs that provide real value.

This sounds fascinating. Can you provide any detail regarding the nature of the diagnosis or problem it identified?

Anecdote but I gave Gemini Pro an image of an individual with Herpes Zoster which the doctor said was something else. Gemini gave the correct diagnosis which allowed for correct treatment and cure.

I don't understand why doctors don't prompt LLMs before saying wrong things. Is it ego?

I can understand for radiology because you need a specialized convolutional network, but for more knowledge based things...

mate the other day chatGPT (enterprise) told me that the kernel 7.0.2 was older than 6.69

you cant trust these toys at all. that doesn't make the useless, just untrustworthy.

(We detached this subthread from https://news.ycombinator.com/item?id=48709121.)

This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief.

It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.

Don’t get me wrong, I think we all agree capabilities will eventually improve (and farther-future capabilities could reasonably surpass experts), but really is unclear if the current transformer architectures with their probabilistic/hallucinatory outputs will plateau before they surpass current experts abilities in all promised fields.

Totally agree. I'm a scientist, and like most scientists I have some specialized skills that most of my colleages don't. AI has empowered them to learn and build things that they might have otherwise needed me for. But there have been quite a few cases where it led them very far down a wrong path. This has started happening way more often in the last few months.*

We've known since the beginning that AIs confidently say incorrect things. But now that they can speak confidently about very complex topics, and mostly say correct things, we are letting our guard down and lots of subtle falsehoods are slipping through.

*In one case, I was able to put things back on track because the AI suggested my colleague talk to me; somehow it figured out we were co-workers.

I see your argument, but it's not exactly news that an expert found a flaw in a popular tool. You could say the same about Wikipedia--experts have tons of issues with it, but Wikipedia still provides value to non-experts. The most likely alternative to Wikipedia for non-experts is simply not trying to learn anything new.

Similarly with LLMs, you can't just write them off entirely because they sometimes provide misleading or incorrect advice. The positive utility maximizing view is to learn when you need to call in an expert. I recently moved in to a new house and have used Claude extensively to figure out basic things (e.g., adjusting the garage door height, how to mount a TV). However, when the HVAC suddenly stopped working, I gave Claude a shot for an hour and tried some non-destructive fixes, but then realized I had to call in an HVAC expert.

> Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

Yes, this is exactly so. AI is able to confidently sound plausible enough to convince laypersons or anyone who isn't very familiar with the subject matter, which is a big part of the mass-appeal "magic" of ChatGPT and other similar tools. It's like having a know-it-all friend (who also makes shit up to bridge their own knowledge gaps).

In many non-advanced non-specialized situations, AI is right enough to be at best useful or at worst not harmful (usually landing in the middle somewhere).

But speaking for myself, in areas where I consider myself quite proficient, I can very easily spot the subtle inconsistencies and naive conclusions that AI responses provide, and I have to guide/steer/correct it a lot to get good results when the subject matter is complex enough.

Last week I went to a highly-specialized tertiary clinic about further treatment for a rare medical condition that I was diagnosed and treated for as a child. The two very specialized doctors I met there confirmed a diagnostic mistake that a specialist had made ten years ago. The only reason I pursued a second opinion, ten years later, was because Google Gemini had explained to me that the specialist ten years ago had performed the wrong type of test for my condition.

Do these LLMs make mistakes? They sure do, I see it all the time. But they can also help people make breakthroughs.

And this isn't the only time that Gemini has helped me diagnose long-term health issues, either.

I am not advocating to trust anything they say blindly, but they can be a great place to form new hypotheses and learn the right terms to look for when you are unfamiliar with a subject.

I may be missing something, but I think it's unclear that the parent poster here is necessarily actually contradicting anything the AI said. It may depend on the exact information the OP wrote to Claude and GPT. The full transcripts would be needed. (Though there is definitely a separate point that a doctor would generally better know all the right questions to ask, while current LLMs may be making certain assumptions.)

The LLM may have, from its "perspective", implicitly thought the OP was telling it that he had strong reason to believe there was no calcification and was not considering the bigger picture of possibly receiving an incomplete/poor assessment from the medical staff. In fact, the issue here may be the LLM overly trusting doctors vs. trusting its own expertise.

> no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information

"Be wowed by the convenience and speed", or merely "take advantage of the mere availability"? What most people find to be damning about expert advice is that they simply can't get it anywhere, at any cost that they can afford.

Seems natural enough. There will always be complexity and nuance that is missed by an AI model or person - the world is just super detailed. The more expertise you have the more you will be aware of that nuance. That doesn't mean the model or person is not useful as a starting point.

I dunno. I know a lot of software engineering experts. AI isn't always right, but neither are the people, and it's getting better and better.

Software is one domain where it excels because of structured training data and simulation environments, so I'm well aware it's better here than other areas.

Still there's somewhere balanced between saying every time it's "insufficient or incomplete or outright misleading" and "just trust AI". AI's a useful source of information/reasoning/research, but know you need to validate it's answers for important decisions.

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading. It is only when you do not know what the AI is being asked to do is it likely you will find the output helpful.

I always recommend people try asking LLMs a lot of questions on something they know first. Programmers should start by asking LLMs to work on a codebase they’re familiar with first.

You’re overstating the problem, though. Even for an expert the LLM will get a lot of things right and can be helpful under a watchful eye.

The real problem is knowing how to identify when it’s on the right track and when you need to correct it, because both cases are presented with the same tone and confidence.

An expert can better identify when the LLM output doesn’t sound plausible. Someone unfamiliar with the topic will think everything it says looks correct.

You're not. This site was also bullish on using LLMs as therapists, which defeats the very point of them, and reflects a lack of knowledge on what exactly therapists do for people.

More on topic: if the article's author arrived at a definitively negative result would this have shown up on HN?

On the flip side of this problem, novel best practices lag the medical standard of care, other human failures like corruption and competing priorities notwithstanding.

For example, we had to advocate for certain practices during the birth of our first child that became routine during our second several years later.

So, neither side is guaranteed correct, doctor or citizen researcher (which did not include LLMs in my case, for the record). The truest answer is also the most useless one, applicable to all fields: it depends.

The real question is: if you embrace being a layman, whom do you trust more: LLMs/the internet or experts, like doctors? I think the answer is pretty clearly experts.

You shouldn’t expect frontier models to work on medical imaging. There is much more that goes into building a medical imaging product. First and foremost is data. Medical imaging datasets are not prevalent one the public internet at the scale necessary to have good performance on medical imaging tasks especially MRI. Also the labels are super noisy.

This is completely different than asking for general medical reasoning which is more derived from papers, public standards and textbooks.

Text exists at the right scale but images don’t.

This is a serious issue for young people I think.

I have seen outputs that look good but the actual content is bad. If you’re inexperienced in a field you can’t see it because AI makes anything look right.

I have gotten very good results with AI but you can’t take the first answer at face value. You need to be suspicious and challenging until you tweak out the right answer over time.

The question is how far is AI off compared to the professional that we have access to. World best experts are not accessible to most of us. :(

No, not anytime someone is an actual expert at anything, AI output appears insufficient. That is why experts in various fields use AI.

Then to say "Aha, but all of that is AI psychosis" makes obviously no sense: Why would we trust experts when they offer critique but not when they say "this is helpful"?

Overall: People are not insane. AI makes mistakes and, often, fails completely. AI also helps them do things better, quicker, increasingly so. The jaggedness of AI is confusing and real.

Well that's part of the problem. AI is not accountable - if you take its advice and hurt yourself, who is responsible?

A real doctor is accountable.

They might both "know" a lot of things but implicitly the party who is accountable is going to be more trustworthy.

And I don't see that going away until AI companies must be licensed for application x and can lose their license / be sued if engaging in malpractice.

>I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

media is awash at the moment with experts chiming in to support AI, saying their fields are being revolutionized, etc.

it seems unsurprising to me that the laymen opinion would follow the loudest media trumpets.

AI is an expert in everything you are not.

Yes. The PM’s “with AI I know enough to be dangerous, haha” means “I’m actually dangerous and I don’t realize”

This is true in broader contexts too. Bunch of experts can't agree on something fundamental which is hard to prove/ disprove, and they have strong opinions on the topic.

AI is much worse.

I came here to post this as my experience. AI is magical when I apply it to something I know nothing about. It far exceeds my expectations every single time. I know nothing, but here is a report with animated graphics explaining exactly what I asked it to explain!

In fields where I'm an expert... it makes a lot of silly mistakes that are annoying and I feel like they would just cascade if I didn't correct them early. (I still think it's a net win, but... I watch it and it watches me, and we both do better work. I'd even apply the "magical" adjective when it does stuff I hate but know how to do, like edit Helm charts. What would normally be 20 minutes of me griping about YAML indentation is just a correct diff in seconds. I'll take it!)

So with that in mind, I tend to distrust output that I can't verify. If a doctor was recommending surgery and I thought the plan was too aggressive, I'd get a second opinion. I don't expect Claude Code to have much medical diagnostic ability, as that is really not what the model is trained for, and I know how it performs on work that it's trained and fine-tuned for. That is not to say the output is wrong and that it can't have diagnostic value, just that I personally wouldn't feel safe trusting it. Wrap up the same model with fine-tuning in the domain and a harness that reminds Claude to do a lot of sanity checks, perhaps with a human in the loop to guide it back onto the rails when it gets hyperfixated on something that doesn't matter? That could very much be a useful AI product.

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading.

AI isn't even the first instance of this phenomenon, news articles are like this as well.

https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

> Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading.

The term for when the press "gets it wrong" is Gell-Mann Amnesia (https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect).

In that case, when you have personal knowledge of the facts, or know the specific domain area, you can see where the reporter mixed things up.

AI is no different, it's just a bunch of matrix math substituting for "the reporter" regurgitating what it was previously told. So the Gell-Mann Amnesia effect would apply just the same. If you have domain knowledge, you immediately see where the AI got it wrong. When you do not have domain knowledge, you have less chance of seeing where the AI was wrong.

Your instinct is correct, and in a lot of cases it's true. However, I've heard from enough doctors by now (a cardiologist, psychiatrist, and epidemiologist/former physician) that they use medical LLMs and find them extremely helpful, mostly as a way to either bring up knowledge they'd forgotten about or as a way to learn something new and then verify it. I'm extremely skeptical about LLMs in general and the connection to Gell-Mann Amnesia is apt, but I wouldn't necessarily write them off completely like that. There are experts using the models that find them genuinely helpful in their field.

TFA doesn’t actually state where the bit about shockwave therapy came from and it wasn’t the main point of the article. The concern was about being given useless therapies. The homeopathic analgesic is concerning, at least to me.

I.e. nothing this radiologist said was related to the LLM’s advice.

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

AI assistant are industrializing the Gell-Mann amnesia effect.

>AI output appears insufficient or incomplete or outright misleading

It has been like this since the rise of "AI". The only people enthusiastic about it are usually the ones hoping to make a profit in one way or another.

It's like reading news articles. Seems reasonable until you read an article about something you know, then you see how wrong they can be.

LLM is not necessarily an expert system. Once there are expert systems for law, healthcare, accounting, governance…

https://en.wikipedia.org/wiki/Expert_system

We're past the point of Gell-Mann amnesia. This is full blown Gell-Mann psychosis.

> This is itself alarming to me, but no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information.

Welcome to the club? This new awareness you've found over the true quality of LLM based GenAI output has been what "all the haters" have been mad about for-ever. That the output of LLMs are clearly defective, and merely have found a cute trick towards making humans think they're less defective than they are actually measured to be.

And the corresponding anger and frustration to push the risks of genai output out onto others, while also aggressively pushing it as a feature you should be using already. You're behind don't you know, and whatever other lie I have to tell to trick you into enough FOMO to pay me 200USD/mo so I can sell FOSS back to you.

An LLM can only output the mean next likely token, and then add a bunch of extra noise on top of that so it feels interesting and not repetitive. None of this is new, the problem is, 50% of humans are below the mean, but have no idea. So when an LLM tells them some lie: well, it sounds so helpful! It's impossible for someone who sounds this helpful to lie to me, liars never sound confident! It must be PERFECT! I'm gonna tell everyone how perfect it is. so the bottom 0-33% think LLMs are fantastic tools that make nearly 0 mistakes in comparison to the bottom 33%. 33-66%-ish aren't sure, some times it's great, but it will make that random mistake sometimes, but I can catch most (or all of them depending on ego). and the 66%+ are angry about how many people are getting tricked by something so obviously low quality, or are lucky enough to not have to care.

This is natural and even logically expected. It's just Gell-Mann amnesia in action. The world has more people spouting on things than it has people knowledgeable in said things.

Apply that to the Internet at large, and realize where LLMs got their training. They're basically ConfidentlyIncorrect personified.

what is happening is that the gap between what the experts and AI know is getting smaller each year. this year sure radiologists are mocking AI's ability to interpret MRI results, but they are a lot better at that this year than last. In five years perhaps radiologists will truly appreciate AI, but I am not holding my breath because radiologists are notoriously slow to adapt to changes in medical science compared to other specialists like anesthesiologists or surgeons

That study seems to be confounding factors and rushing to a questionable conclusion.

A very plausible explanation for the adenoma detection rate to have gone down is simply that its prevalence went down among the population in the second three-month period.

This was not a randomized trial. Concluding that "AI usage degrades physicians' skills" is questionable at the very least.

I don’t even trust AI to proofread my emails.

In the US, this is standard advice. I note that the OP is in Germany. Maybe they do things differently, there.

It’s an odd field, obviously it’s in high demand for diagnostics and anytime you have to do an xray, MRI, etc you have to wait hours for one to become available.

I think they’re artificially stunting the field to raise their wages. For example in my city the medical school only accepts 11 people into the program a year. (With an average graduation rate or 3-5). My niece has been trying for 2 years and finally got in this last year. Even radiology is doing AI assisted diagnostics. Half my MRI’s from this year has Doctor notes and HealthBot (AI) notes attached to them.

~ I’m assuming other schools severely limit their radiology admissions as well. To keep the wages high and the field desirable.

> Was it 2016 when Geoff hinton said that radiology was a dead career?

Funny how the jobs most at risk of automation now are tech jobs.

yeah that is why you would not use a random llm that is not trained in radiology lmfao

diffusion models are probably a better bet for identifying irregular structures

I work at a telemedicine company. We’ve benchmarked a few frontier LLMs on public medical imaging datasets. One test included high-quality and high-consensus otoscopic images. We didn’t anticipate the models to do well on something so niche, but what concerned us was how poorly calibrated the models were.

I know you can’t trust an LLM’s self-assessed “confidence” of a prediction, but I’ve found that confidence can at least be directionally correct for some tasks. For our benchmarks, however, confidence was poorly correlated. What’s worse is that binary classification models (“Do you see $diagnosis in this photo?”) highly influenced the LLM to confidently predict $diagnosis.

I’m concerned for those using LLMs for diagnostics, and getting confidently led to the wrong conclusion.

It makes a lot of sense if you understand how these models work but this was a cool read anyways and studies like this are impotent for curbing the unfortunate fever dream some folks seem to be collectively having about LLM omnipotence

I don’t understand how this is a different result than giving any LLM a task that is not completely grounded? I’ve observed this in coding tasks, if I forget to include a file referred to in the spec, the LLM will just hallucinate a version of it and my results suck. If I give it the file (and really, all the information I claimed it had access to), the task works fine. I fixed this in my pipeline with a prompt that does an extensive grounding analysis to determine if the assets I’m giving it are complete with respect to the spec (and that the spec is grounded as well, ie it doesn’t refer to something that is undefined).

I wonder if the above problem can be fixed similarly? Just ask the LLM to do a conservative grounding analysis before jumping to the main task?

The absolute only thing that matters is if they are provided an image what's the success rate.

But why should I care? If you demonstrated that a model can perform more accurate diagnoses than a doctor, but also it had this strange behavior when no image was presented, why should that deter me from using the model?

I need this but for dentists.

I agree with you for some kinds of images, but not all.

LLMs are the best PDF-to-markdown converters, in my experience. I have a CLI that converts PDF to PNG, then run a background agent to "read" each PNG and write it down as markdown; it works flawlessly even for complex math formulas, it can "translate" complex charts, graphs, and tables into words.

It's slow and arguably expensive compared to traditional OCR, but very effective and precise.

Especially an MRI which is a 3D medium —something current LLMs are very bad at.

I used it on an ankle fracture xray, it was quite useful to make sense of things. But not like a 2nd opinion.

What's wrong with Claude? I've asked it to analyze images and even Opus 4 would perfect nail it.

Interesting. I am also diagnosed with ET (from platelet counts, pathology report noting abnormal platelet morphology, and JAK2 mutation from a marrow biopsy). What made you seek a second opinion?

Do you think you would ever use LLMs in tandem in your practice to help diagnose and treat?

I wish you were my doctor!

This...

Gemini will ask for the specific images it needs to see and show you examples of what each slice will look like.

But unlike the specialists who often come across as abrupt and rush you, Gemini will happily take you on a deep dive and continue to answer all you follow up questions indefinitely.

> standard run the images through a certified compliant LLM?

Because they don't exist, yet.

In the UK MRIs and other imaging systems need two opinions. there has been a move to allow the first opinion to be ML based.

The _problem_ is that you are basically doing grey smudge analysis, and thats fucking hard.

I've been starting to think of LLM as a great tool for "lead generation," borrowing a term from sales. Most of the things it comes up with don't pan out, but in many cases it's things we wouldn't have thought of, or at least not as quickly. This is especially in the context of web service or SAAS outages.

Because they might bias you. And because you have your own brain, training and experience

Not as smart as modern frontier models, but Moondream and Molmo can do that sort of thing.

I do powerlifting and couple years ago, I developed bicep tendinitis on my right arm. Even a tiny bit of weight on it while palm facing up would cause crazy pain. It was funny how I weight from lifting heavy weights to not even being able to carry a plate of food, not being able to press soap dispensers, or give a spot to someone at the gym.

Even a tiny injury can severely cripple us.

I’m wondering how many radiologist want to work all day, then come home and work.

Many can get paid fee-for-service for after hours work, so would probably prefer that.

I am confused why none of the experts weighing in here address this at all. Like I get that AI is generally disliked, but ignoring facts only makes me want to not trust doctors.

> I won't go into the details, but he suggested I get an MRI, which the clinic conveniently had available.

And

> They performed shockwave therapy on my shoulder

(a procedure that may not be effective, but is unlikely to cause any harm)

Its not just about LLM's being better, its about people not trusting DR any more: https://www.physiciansweekly.com/post/the-erosion-of-trust-i...

If we want to fault the article for anything it's that he didnt take that information and go get a 2nd opinion from someone who IS more informed.

I think this draws too strong a line between the matrix-math core and the harness that uses it. Those harnesses undoubtedly were built with purpose and the systems fail to achieve that goal. Common usage says the the DMV can make mistakes, like any systems, despite the DMV itself not being a person (and it is common to allege large organizations make mistakes even when no specific individual is making an identifiable mistake). This isn't person-language it's systems/purpose-language.

Or you have an interval?

> But are you all forgetting that they literally injected a homeopathic drug on the author?

I'm no fan of pseudoscience either, but this is where things get blurry. The placebo effect is real even if patients are aware of it. If you give a patient a homeopathic drug while informing them of potential side effects (if any), and then they feel better, have you hurt them? Or have you helped them?

I personally have no interest in trying homeopathic medicines, but the reality is that many patients do take these and are adamant they help. As long as any risks are communicated and there are no serious side effects, it's difficult to make an argument against their use in patients who report a subjective benefit.

It gets worse when they challenge your solutions by feeding it back into the LLM and sending the response on to you, arguing with an LLM is exhausting, arguing by proxy with a human parroting its responses is excruciating.

On the plus side when they do this they can't flood your calendar with those "quick chat" meetings because they know they won't be able to hold a conversation on the issue beyond the first minute.

I've seen coworkers do this to each other when their expertise is in different domains.

I find that AI can be incredibly useful, but just text dumping its output into a conversation feels insulting.

True, but this was a problem long before AI (read this article, met this guy at a conference who told me x, my boss said blah)

AI probably exacerbates it but crappy managers exist regardless

Sometimes I get a lot of "Do you want me to work up how the UI will look."

They give me what they'd like the UI to look like, but none of the actual content fits outside the one situation they're thinking of.

¯\_(ツ)_/¯

Thankfully where I work now everyone is good about taking no for an answer.

Fight fire with fire. It's over the top passive aggresive, but it works. Whenever I get a JIRA ticket that was clearly AI generated and is 10x too many words, I tell Claude to respond to that ticket with my actual real opinion or suggestion, but make it 10x more words.