How do you suggest to deal with Gemini? Extremely useful to understand whether something is worrying or not. Whether we like it or not, it’s a main participant to the discussion.
Apparently we should hire the Guardian to evaluate LLM output accuracy?
Why are these products being put out there for these kinds of things with no attempt to quantify accuracy?
In many areas AI has become this toy that we use because it looks real enough.
It sometimes works for some things in math and science because we test its output, but overall you don't go to Gemini and it says "there's a 80% chance this is correct". At least then you could evaluate that claim.
There's a kind of task LLMs aren't well suited to because there's no intrinsic empirical verifiability, for lack of a better way of putting it.
This "random output machine" is already in large use in medicine so why exactly not? Should I trust the young doctor fresh out of the Uni more by default or should I take advises from both of them with a grain of salt? I had failures and successes with both of them but lately I found Gemini to be extremely good at what it does.
The "well we already have a bunch of people doing this and it would be difficult to introduce guardrails that are consistently effective so fuck it we ball" is one of the most toxic belief systems in the tech industry.
> This "random output machine" is already in large use in medicine
By doctors. It's like handling dangerous chemicals. If you know what you're doing you get some good results, otherwise you just melt your face off.
> Should I trust the young doctor fresh out of the Uni
You trust the process that got the doctor there. The knowledge they absorbed, the checks they passed. The doctor doesn't operate in a vacuum, there's a structure in place to validate critical decisions. Anyway you won't blindly trust one young doctor, if it's important you get a second opinion from another qualified doctor.
In the fields I know a lot about, LLMs fail spectacularly so, so often. Having that experience and knowing how badly they fail, I have no reason to trust them in any critical field where I cannot personally verify the output. A medical AI could enhance a trained doctor, or give false confidence to an inexperienced one, but on its own it's just dangerous.
No, I'm not asking you spend $150, I'm providing you the evidence your looking for. Mayo Clinic, probably one of the most prominent private clinics in the US, is using transformers in their workflow, and there's many other similar links you could find online, but you choose to remain ignorant. Congratulations
The existence of a course on this topic is NOT evidence of "large use". The contents of the course might contain such evidence, or they might contain evidence that LLM use is practically non-existent at this point (the flowery language used to describe the course is used for almost any course tangentially related to new technology in the business context, so that's not evidence either).
But your focus on the existence of this course as your only piece of evidence is evidence enough for me.
Focus? You asked me for an evidence. I provided you with the one. And with the one which has a big weight on it. If that's the focus you're looking for then sure. Take it as you will, I am not here to convince anyone in anything. Have a look in the past to see how Transformers have solved the long standing problems nobody believed they are tractable up to that point.
LLM is just a tool. How the tool is used is also an important question. People vibe code these days, sometimes without proper review, but do you want them to vibe code a nuclear reactor controller without reviewing the code?
In principle we can just let anyone use LLM for medical advice provided that they should know LLMs are not reliable. But LLMs are engineered to sound reliable, and people often just believe its output. And cases showed that this can have severe consequences...
There's a difference between a doctor (an expert in their field) using AI (specialising in medicine) and you (a lay person) using it to diagnose and treat yourself. In the US, it takes at least 10 years of studying (and interning) to become a doctor.
Even so, it's rather common for doctors to not be albe to diagonise correctly. It's a guessing game for them too. I don't know so much about US but it's a real problem in large parts of the world. As the comment stated, I would take anything a doctor says with a pinch of salt. Particularly so when the problem is not obvious.
This is really not that far off from the argument that "well, people make mistakes a lot, too, so really, LLMs are just like people, and they're probably conscious too!"
Yes, doctors make mistakes. Yes, some doctors make a lot of mistakes. Yes, some patients get misdiagnosed a bunch (because they have something unusual, or because they are a member of a group—like women, people of color, overweight people, or some combination—that American doctors have a tendency to disbelieve).
None of that means that it's a good idea to replace those human doctors with LLMs that can make up brand-new diseases that don't exist occasionally.
It takes 10 years of hard work to become a profound engineer too yet it doesn't prohibit us missing the things. That argument cannot hold. AI is already wide-spread in medical treatment.
An engineer is not a doctor, nor a doctor an engineer. Yes, AI is being used in medicines - as a tool for the professional - and that's the right use for it. Helping a radiologist read an X-Ray, MRI scan or CT Scan, helping a doctor create an effective treatment plan, warning a pharmacologist about unsafe combinations (dangerous drug interactions) when different medications are prescribed etc are all areas where an AI can make the job of a professional easier and better, and also help create better AI.
Nobody can (and should) stop you from learning and educating yourself. It however doesn't mean just because you can use Google or use AI, you think you can become a doctor:
Educating a user about their illness and treatment is a legitimate use case for AI, but acting on its advise to treat yourself or self-medicate would be plain stupidity. (Thankfully, self-medicating isn't as easy because most medication require a prescription. However, so called "alternate" medicines are often a grey area, even with regulations (for example, in India).
With robust fines based on % revenue whenever it breaks the law, would be my preference. I'm nit here to attempt solutions to Google's self-inflicted business-model challenges.
If it's giving out medical advice without a license, it should be banned from giving medical advice and the parent company fined or forced to retire it.
As a certified electrical engineer, the amount of times googles LLM suggested a thing that would have at minimum started a fire is staggering.
I have the capacity to know when it is wrong, but I teach this at university level. What worries me, are the people who are on the starting end of the Dunning-Kruger curve and needed that wrong advice to start "fixing" the spaces where this might become a danger to human life.
No information is superior to wrong information presented in a convincing way.
Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
A few months ago in a comment here on HN I speculated about the reason an old law might have been written the way it was, instead of more generally. If it had been written without the seemingly arbitrary restrictions it included there would have been no need for the new law that the thread was about.
A couple hours later I decided to ask an LLM if it could tell me. It quickly answered, giving the same reason that I had guessed in my HN comment.
I then clicked the two links it cited as sources. One was completely irrelevant. The other was a link to my HN comment.
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me?
I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone must have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had.
Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
Yeah, this is by far the biggest value I've gotten from LLMs - just pointing me to the area of literature neither me nor any of my friends have heard of, but which have spent a decade running about the problems we're running into.
In this case, all that matters is that the outputs aren't complete hallucination. Once you know the magic jargon, everything opens up easily with traditional search.
And that becomes obvious when you go to look for it. These work best in situations where false positives have low cost/impact, but true positives are easily verifiable and have high impact. in other words, problems in epistemological p-space. :)
I had a similar thing happen to me just today. A friend of mine had finished a book in a series. I have read the series but it was almost 10 years ago, and I needed a refresher with spoilers, so I went looking.
Well, some redditor had posted a comparison of a much later book in the series, and drawn all sorts of parallels and foreshadowing and references between this quite early book I was looking for and the much later one. It was an interesting post so it had been very popular.
The AI summary completely confused the two books because of this single reddit post, so the summary I got was hopelessly poisoned with plot points and characters that wouldn't show up until nearly the conclusion. It simply couldn't tell which book was which. It wasn't quite as ridiculous as having, say, Anakin Skywalker face Kylo Ren in a lightsaber duel, but it was definitely along those same lines of confusion.
Fortunately, I finished the later book recently enough to remember it, but it was like reading a fever dream.
Yeah, it happened recently for a kubernetes resource. I was searching for how to do something, and Google AI helpfully showed me a kubernetes resource that was exactly what I needed, and was designed to work exactly how I needed it.
Sadly, the resource didn't actually exist. It would have been perfect if it did, though!
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
At some point in time when asked how many Kurdish people live in Poland, Google's AI would say that several million, which was true, but only in a fantasy world conjured by a certain LARP group, who put a wiki on fandom.com.
Or you can take the alternative approach, where Microsoft's own "Merl" support agent says it knows anything to do with Minecraft, and then replies to basically any gameplay question with "I don't know that".
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it.
Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for.
I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them...
https://deadstack.net/cluster/google-removes-ai-overviews-fo...
Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
When I look at the US, the symptom -> diagnosis hypothesis is not anywhere near the most expensive bit. If you have a medical issue and AI works effectively for this then it saves you maybe one trip to your GP. Your insurance probably still requires your GP to provide a referral to a specialist. If insurance companies allow for AI to be used in place of a referral then you save this trip. But you still need all of the stuff to confirm a diagnosis. And you still need all of the treatment.
If you don't have a medical issue and an AI system tells you this then you save yourself a trip to a specialist and the associated diagnostic tests. Again, this saves a bit of money but is nowhere near the bulk of medical expenses. And it has to be able to do this without any diagnostic testing, just based off of your reported symptoms.
Even if AI diagnosis works flawlessly we save a bit of money but absolutely do not revolutionize the cost of the industry.
Salaries for healthcare workers make up only a small portion of expenditures. You do not want to avoid a trip to your GP for an AI system.
It'll be great at first while in development. But when profits need to be generated, seeing a specialist will get harder. There will be less wiggle room. I predict we will see more GP utilization.
I mean, if we're talking Christensenian disruption, that happens in neglected markets rather than currently dysfunctional ones. There's no shortage of actors wronging money out of health care so there's not a disruptable space per se.
It's really just not there yet. I've been in medical school for >3 years now and have been using the latest models with good prompting. They have gotten much better, but I still see misses that my classmates would easily catch. This is not acceptable in healthcare. It's certainly not getting 100% on all my assignments, which are a step below the complexity of real-world clinical practice.
Before medical school, I was not so sure of the quality of your average doc. Now having spent a year in clinical practice across various settings, I am extremely reassured. I can say with certainty that a US trained doctor is miles ahead of AI right now. The system sucks really bad though and forces physicians to churn patients, giving the impression that physicians don't pay attention/don't care/etc.
because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
Physician compensation is around 9% of healthcare spending. The number of non-physician providers (NPs, PAs and specialists like physical therapists and podiatrists) has also exploded over the last 20 years. We have far more overall providers per capita than we did 20 years ago.
I don't think physician compensation per se is a good metric for capturing the effect of lack of providers, because some of the increased costs are due to the bottlenecks in the services per se, in terms of procedure costs and types of procedures offered. I also don't think the number of providers per se under the current regime, without deregulation or reregulation of practice boundaries, is representative of what would happen if there were changes in those boundaries. Adding more optometrists 5 years ago isn't the same as changing what they're allowed to do. It also doesn't address what cost increases would have been without an increase in the number of providers.
9% might also seem pretty big to me if it's out of all spending and doesn't include other provider compensation? What if overall healthcare costs went down, but physician compensation stayed the same? Would that then be a problem because it was an increased proportion of the total costs — fat left to be trimmed, so to speak?
There are many problems that don't have anything to do with providers per se, but I also don't think you can glean much by extrapolating to more of the same, especially compensation per se.
If lack of physicians is leading to an increase in costs, you’d expect to see physicians capturing a large part of that increase. There are situations where that wouldn’t hold, but it requires moving away from the simplest explanation that fits the data.
We don’t have to extrapolate from physician compensation though. We know that providers per capita have increased, but costs have continued to skyrocket. Therefore a lack of providers is not the immediate cause of the increase.
In addition to increasing the number of providers, the scope of practice for non-physician providers has almost universally increased.
All of this doesn’t prove that increasing the number of physicians wouldn’t lower costs some amount, but it does show that the increases over the last 20-30 years requires some other explanation.
Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
I sympthatise with the argument. We should test it against real world data.
Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
Dental and vanity surgeries aren't happening in a vacuum. There are baseline costs eg. anesthesia, recovery medications, medical machinery etc which are all bloated due to the rest of industry not being under price pressure (rising tide lift all boats)
It's similar to how AI data center buildout race is raising the prices for consumer electronics in 2026 and beyond. The suppliers have no incentive to sell lower cost products to tiny niche
I just looked it up, and apparently health care costs for pets has gone up in price even faster than for humans.
Pets typically don't have medical insurance, and any insurance that does exist there has a radically different regulatory regime than for humans.
Since 1980 for the US:
CPI has gone up by 3.16% on average per year (x4.17 in total). Human healthcare costs by 4.9% per year (x8.96 in total). And pet healthcare costs by 6.49% (or x17.87 in total).
Technological advancement seems like a pretty big culprit here? An MRI wasn't available at any price in 1982 so the fact that it's available now makes it look like "imaging prices have gone up" even though the imaging that was available in 1982 is cheaper now than it was then.
The uninsured medical market is actually quite efficient. e.g. lasik surgery can actually be done for very reasonable rates with full price transparency.
Lasik has gone down in real price over the years, yes.
But dental and vanity cosmetic surgery have gone up by that metric. Dental is less covered by insurance for most people. Vanity cosmetic insurance is covered for almost no one.
Vet care for pets has gone up a lot more than healthcare for humans.
This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
The real source of high medical costs is the entity that sets the hospital bill in the first place.
The explanation is much simpler than people want to admit, but emotionally uncomfortable: doctors and hospitals are paid more than the free market would otherwise justify. We hesitate to say this because they save lives, and we instinctively conflate moral worth with economic compensation. But markets don’t work that way.
Economics does not reward people based on what they “deserve.” It rewards scarcity. And physician labor is artificially scarce.
The supply of doctors is deliberately constrained. We are not operating in a free market here. Entry into the profession is made far more restrictive than is strictly necessary, not purely for safety, but to protect incumbents. This is classic supply-side restriction behavior, bordering on cartel dynamics.
We see similar behavior in law, but medicine is more insidious. Because medical practice genuinely requires guardrails to prevent harm and quackery, credentialing is non-negotiable. That necessity makes it uniquely easy to smuggle in protectionism under the banner of “safety.”
The result is predictable: restricted supply, elevated wages, and persistently high medical costs. The problem isn’t mysterious, and it isn’t insurance companies. It’s a supply bottleneck created and defended by the profession itself.
Insurance companies aren't innocent angels in this whole scenario either. When the hospital bill fucks them over they don't even blink twice when they turn around and fuck over the patient to bail themselves out. But make no mistake, insurance is the side effect, the profession itself is the core problem.
> This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
They absolutely do not.
They have their profit levels capped at 15% by law and regulation. That means if the insurer wants more absolute dollars of profit, prices must go up.
It also means that if they push prices down they necessarily have less funding to administer those plans, even if the needs are the same (same number of belly buttons, same patient demographics and state of health).
As you note there's also other variables, but this claim: "Insurance companies are structurally incentivized to minimize payouts across the board" is absolutely and categorically not so.
Physician reimbursement is only ~9% of national healthcare expenditures.
I tell you this with certainty as a 3rd year medical student: If physician wages go down and tuition stays as is, no one will do this. Intrinsic motivation to help people evaporates as soon as you see how enshittified healthcare in the US has become.
I do agree that medical school is far too restrictive to get into (For MD schools at least). However, if you want to make medical school easier to get into: Where will all those students rotate at for their clinical years? There aren't enough spots in hospitals to jam students in.
Stop taking aim at the people that sacrifice so much to help you. Take aim at the real drivers of healthcare expenditures: administrative bloat.
>Where will all those students rotate for their clinical years? There aren’t enough hospital slots.
This is a policy fiction. Residency slots are capped by federal law, not by hospital capacity. The Balanced Budget Act of 1997 froze Medicare-funded residency positions, and despite modest expansions decades later, the cap remains largely intact. Teaching hospitals routinely report excess clinical volume relative to trainee supply. The bottleneck is artificial and regulatory, not logistical.
>Stop taking aim at people who sacrifice so much to help you. The real cost driver is administrative bloat.
This framing collapses under scrutiny. Administrative bloat is real and well-documented, but pretending physician incentives are irrelevant requires willful blindness. Numerous studies show that U.S. physicians earn multiples of their OECD peers while delivering no commensurate advantage in outcomes. Many doctors are motivated by altruism, but many are also motivated by status, income, and professional gatekeeping—normal human incentives in a high-prestige, high-pay profession.
Further, high patient throughput is not an accident. Fee-for-service reimbursement structurally rewards volume over care quality. Seeing 20–30 patients a day is not a moral failure of individual doctors, but it does predictably lead to burnout, emotional detachment, and assembly-line medicine. Incentives shape behavior. Ignoring that is not compassion, it’s denial.
>Physician reimbursement is only ~9% of national healthcare spending.
That statistic is repeatedly used as a rhetorical shield, and it shouldn’t be. Cost systems do not fail because of a single oversized line item; they fail because multiple protected constituencies simultaneously extract rents while deflecting blame. Administrative overhead, defensive medicine, pharmaceutical pricing, hospital consolidation, reimbursement incentives, and physician compensation are jointly optimized for revenue, not outcomes.
Nine percent of a multi-trillion-dollar system is not trivial. More importantly, physician compensation is not isolated—it drives downstream costs through referral patterns, test ordering, procedure rates, and resistance to scope-of-practice reform. Treating physicians as a sacred class exempt from economic critique is precisely how you end up with a system that is unaffordable, unaccountable, and structurally resistant to reform.
If the argument is “9% is too small to question,” then by that logic no component is ever large enough to examine in isolation, which is how dysfunctional systems persist indefinitely. Real reform requires abandoning moralized narratives and admitting the obvious: healthcare costs are the product of aligned incentives across many actors, and physicians are not magically outside that system simply because the story is uncomfortable.
The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here.
And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
If we could get healthcare to that level, it would be great.
For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.
> Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
Google searches cost many billions of dollars: your confusion is because the customer isn’t the person searching but the advertisers paying to influence them. Healthcare can’t work like that not just because the real costs are both much higher and resistant to economies of scale but, critically, there aren’t people with deep pockets lining up to pay for you to be healthy. That’s why every other developed country sees better results for less money: keeping people healthy is a social good, and political forces work for that better than raw economic incentives.
We know that from observing evidence such as how much the government pays out in welfare to Wal-Mart employees.
Customers continue shopping there because human beings are typically incapable of accepting a short-term loss (higher price) for a long-term gain (product lasts more than three uses).
> We know that from observing evidence such as how much the government pays out in welfare to Wal-Mart employees.
That's a weird metric. If tomorrow Wal-Mart laid off all employees and replaced them with robots, they would surely be worse off, but by your metric Wal-Mart would look less evil?
> Customers continue shopping there because human beings are typically incapable of accepting a short-term loss (higher price) for a long-term gain (product lasts more than three uses).
I mean, I could tell you are disingenuous from the get-go, so it is not surprising you would take an off-the-cuff metric which is accurate right now and invent a strawman scenario where I might continue to use it beyond a point where it makes sense.
Likewise, I would not use my flippant 3 times metric regarding durability to cover the quality of produce.
Well, feel free to propose a different metric for seeing how Wal-Mart is evil. And instead of Wal-Mart firing all employees, we can have a look at the more plausible scenario of Aldi vs Wal-Mart. Aldi runs a different labour model: their shifts are much more intense but pay more. Their model can't and doesn't utilise the kind of older and less fit people that Wal-Mart employs.
You have to look at the counterfactual of what these people would do, if Wal-Mart weren't around. You seem to implicitly assume that they'd be getting higher paying jobs somewhere else (so they wouldn't have to rely on welfare)? If so, what's stopping those people from switching to these better jobs right now, even while Wal-Mart is still around?
And sure, let's disregard how many times you can eat your groceries. That was a cheap shot. However I think quality vs price trade-off is something customers have to make for themselves anyway. Who am I to judge their choices?
And Google search, a service on the level of a public utility, has been degrading noticeably for years in the face of shareholders demanding more and more returns.
Comparing something to a public utility is not me saying it's literally a public utility. Google runs a monopolistic service that is essential to a lot of our public life, in a segment that has high cost of entry and infrastructure cost. They make the service worse to make more money. It should be a regulated utility like electricity or railroads, we should have a public alternative like the post office is to UPS, or it should be nationalized. The situation gets more dire when you consider their browser monopoly.
Other search engines exist. Bing is right there, and Microsoft is more than willing to eat the high cost of entry and infrastructure cost.
> It should be a regulated utility like electricity or railroads, we should have a public alternative like the post office is to UPS, or it should be nationalized.
I agree that electricity and railroads should be regulated like Google Search.
It's really weird that snail mail in the US is a government monopoly. When even social democratic Germany managed to privatise them.
> The situation gets more dire when you consider their browser monopoly.
Don't a lot of people in the US use iPhones? They don't ship with Chrome as the default browser, do they?
(And yes, Safari is built on top of the same open source engine as Chrome. But you can hardly call using the same open source project a 'monopoly'. Literally anyone can fork it.)
The existence of few competitors is not proof that monopolistic power doesn't exist and isn't being leveraged. Saying Google isn't monopolistic is being willfully wrong. You're more wrong when we look at the browser market, and Google has lost anti-trust suits on this very topic in the past couple years.
A public mail service is required by our constitution. It's cheaper than the private options and often the only option for many rural areas. It's not a monopoly.
> A public mail service is required by our constitution.
Where does it say so in your constitution? All I can find is the postal clause which Wikipedia summarises as follows, but whose full text isn't much longer:
> Article I, Section 8, Clause 7, of the United States Constitution, the Postal Clause, authorizes the establishment of "post offices and post roads"[1] by the country's legislature, the Congress.
The Postal Clause certainly allows the government to run a public postal service, but I don't see how the constitution _requires_ it. It doesn't even require the federal government to regulate postal services, it merely allows it.
Perhaps I missed something?
> It's cheaper than the private options and often the only option for many rural areas.
If you want to subsidise rural areas, I would suggest to do so openly, transparently and from general taxation. At least general taxation is progressive etc. Instead of just making urban folks pay more for their mail, whether they be rich or poor.
I would also suggest only subsidising poor rural areas. Rich rural areas don't need our help.
IDK, the owners of retail clothing chains buy yachts and yet that sector is jaw-droppingly efficient at delivering clothes to people. Executives can be annoying tools but I don't think their pay is the problem.
Shame those same owners have plenty of money for buying yachts and not enough money to pay their sweat-shop employees a living wage or to provide decent work conditions. Executives don't "earn" huge paychecks, they merely exploit others by figuring out a way to not pay them their worth.
Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.
> large insurance pools that must span age groups and risk groups.
What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.
> Mandate at least decent minimal coverage standards
I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)
Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.
There are a lot of not-for-profit insurance companies and they aren't noticably cheaper, though I'm not in HR and they may well be cheaper for the employer.
Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?
Profit isn't even a big part of the overall revenue.
> Mandate at least decent minimal coverage standards
I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.
> and large insurance pools that must span age groups and risk groups.
Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
> Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
There's insurance which allows you to convert an uncertain danger into a known payment. And then there's welfare and redistribution.
By all means, please run some means testing and give the poor and sick or disabled extra money. Or even just outright pay their insurance premiums.
But please finance that from general taxation, which is already progressive. Instead of effectively slapping an arbitrary tax on healthy people, whether they be rich or poor. And please don't give rich people extra stealth welfare, just because they are in less than ideal health, either.
Just charge people insurance premiums in line with their expected health outcomes, and help poor people with the premiums using funds from general taxation. (Where poor here means: take their income and make an adjustment for disability etc.)
We _want_ the guy who loses 5kg and gives up smoking to get lower insurance premiums. That's how you set incentives right.
> The more diverse the pool, the lower the risk.
No. The diversification comes from the insurance company running lots of uncorrelated contracts at the same time and having a big balance sheets. For that, it doesn't matter whether it's a pool of similar insurance contracts, or whether they have bets on your insurance contract, and on the price of rice in China, and playing the bookie on some sports outcomes etc. In fact, the more diversified they are, the better (in principle).
But that diversification is completely independent of the pricing of your individual insurance contract.
Have a look at Warren Buffett's 'March Madness' challenge, where he famously challenges people to predict all 67 outcomes of some basketball games to win a billion dollar. Warren Buffet ain't no fool: he doesn't need a pool, he can price the risk of someone winning this one off challenge.
This is going to pretty rapidly devolve into cheap for healthy and insanely expensive for those that aren’t. Genetic propensities will be a lifelong financial burden. Cancer patients will get priced out and die.
If you want to subsidise these unfortunate people, please just do so openly. Don't be all sneaky, sneaky about it by banning accurate risk assessments.
In any case, what you are saying is only true, if you buy your health insurance second to second on the spot market.
Insurance companies are more than happy to enter long running contracts, where you both agree today on (the algorithm for) the premiums for the next twenty years or even until the rest of your life. That's pretty common with life insurance and disability insurance already.
The above already exists, but if you allow some speculation: you could even envision people buying insurance for their kids before conceiving them. That way you don't have to worry about pre-existing conditions.
(Well, if the parents already have heritable conditions that would make the kids more likely to have expensive medical problems, those would push up their premiums. But then: perhaps these people should think twice about burdening a potential kid with these issues.
Compare how in Cyprus where sickle cell anemia is prevalent, even the Catholic church demand you get screened, before they'll marry you.)
If you really want specialised in-kind welfare, you can get people a voucher for the catastrophic version of 'unconceived baby insurance'.
Basically, you can buy insurance against insurance premiums being expensive.
Disruption, yes, in the sense that the current system needs to be overhauled. But this is a space that's frequented by the SV and VC space and "disruption" has very different connotations, usually in the realm of thought that suggests some SV-brained solution to an existing problem. In some edge cases like Uber/Lyft, this upending of an existing market can yield substantial positive externalities for users. Other "heavy industry" adjacent sectors, not so much. Healthcare and aviation, not so much.
Even SpaceX's vaunted "disruption" is just clever resource allocation; despite their iterative approach to building rockets being truly novel they're not market disruptors in the same way SV usually talks about them. And their approach has some very obvious flaws relative to more traditional companies like BO, which as of now has a lower failure-to-success ratio.
I don't think you'll find many providers clamoring for an AI-assisted app that hallucinates nonexistent diseases, there are plenty of those already out there that draw the ire of many physicians. Where the industry needs to innovate is in the insurance space, which is responsible for the majority of costs, and the captive market and cartel behavior thereof means that this is a policy and government issue, not something that can be solved with rote Silicon Valley style startup-initiated disruption; that I would predict would quickly turn into dysfunction and eventual failure.
Enshittification has done a lot of damage to the concept of "disrupting" markets. It's DOA in risk-averse fields.
The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
I'm not sure why you would think I would blurt that out on the internet.
And are you sure it's giving you good info? "AI" is famously subject to hallucinations, so you may not be getting the "good info" you think you're getting. Be careful with "AI", it's not an all-seeing-all-knowing infallible oracle.
in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be
"confabulation", though, seems perfect:
“Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.”
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
The AI summaries clearly aren’t bad. I’m not sure what kind of weird shit you search for that you consider the summaries bad. I find them helpful and click through to the cited sources.
No sane, in the loop person would believe Google reviews anything, at least not with a human. I bet they have another AI reviewing the answers, and the only time a human enters the loop is when this second AI needs to be restarted.
Tangent, but some people I know have been downloading their genomes from 23andme and asking Gemini via Antigravity to analyze it. "If you don't die of heart disease by 50, you'll probably live to be 100."
23&Me data is simply not accurate enough to make reasonable predictions about health outcomes.
I have a whole genome and nothing Google has built has been able to do anything useful with it, medically speaking. I could use DeepVariant to re-map all the raw reads, it would only slightly increase the accuracy of the estimate of my genome sequence. When I met with genetic counselors, they analyzed my genome and told me I had no known markers for any disease (and they also told me they Google all the unique variants that show up in the report).
(for what it's worth, I literally went to work at Google to improve their medical/health/genomics research, and after working on it a few years I concluded that the entire genomics field is about 90% fantasy. If you want actionable data, there are a small number of well-regulated tests that can help in a limited set of circumstances, but those aren't whole genome tests).
As accurate as our knowledge of genetics, which is not very outside of the identified set of pathological genes associated with hereditary disorders.
Your genome is very complex and we don’t have a model of how every gene interacts with every other and how they’re affected by your environment. Geneticists are working on it, but it’s not here yet.
And remember that 23andMe, Ancestry, and most other services only sequence around 1% of your genome.
Part of genetics is pattern matching, and last time I checked I still can't find a model that can correctly solve hard Sudokus (well, assuming you don't pick a coding model that writes a Sudoku solver.. maybe some of them are trying to do genetics by doing correct algorithms), a trivial job if you write a program that is designed to do it.
Are you asking for you in particular? It's certainly not accurate in general that anyone that made it to 50 is likely to live to 100.
One I heard was if you make it to 80 you have a 50% chance to make it to 90. If you make it to 90 you have a 50% chance to make it to 95. From 95 to 97.5 again 50% chance. That for the general population, in a 1st world country though, not any individual.
The assessment he got was "you have supergenerian genes, but also this one that's really sensitive to heart disease. If you can keep you plaque in check, you're almost invincible (to other popular causes of mortality)."
It cited whatever gene it said he had that made that so.
But only for some highly specific searches, when what it should be doing is checking if it's any sort of medical query and keeping the hell out of it because it can't guarantee reliability.
It's still baffling to me that the world's biggest search company has gone all-in on putting a known-unreliable summary at the top of its results.
Different levels of capabilities. The summary feature in google uses a quick and inaccurate AI model. Were it to be a heavier model, we wouldn’t have this problem.
We would still have this problem. The heavier models make mistakes at too high a rate vs. a physician. Especially on imaging data. Real world data and patient presentations often deviate from the textbooks they are trained on.
That's a different class of problem. It will do just fine on text based queries spanning a few pages. Probably better than the average physician (average over all countries).
I do agree that LLM's are not there yet in the image part.
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy.
Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications.
I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
the problem isn't that search engines are polluted; that's well known. The problem is that people perceive these AI responses as something greater than a search query; they view it as an objective view point that was reasoned out by some sound logical method -- and anyone that understands the operation of LLMs knows that they don't really do that, except for some very specific edge examples.
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk"
Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"
I'm telling you all this as a medical student that has used the latest and greatest models with proper prompting for the past 3 years in school:
There are a ton of misses. Especially on imaging. LLMs are not ready for consumer-facing health information yet. My guess is ~ 3-5 years. Right now, I see systems implementing note writing with LLMs, which is hit or miss (but will rapidly improve). Physicians want 1:1 customization. Have someone sit with them and talk through how they like their notes/set it up so the LLMs produce notes like that. Notes need to be customized at the physician level.
Also, the electronic health records any AI is trained on is loaded to the brim with borderline fraud/copy paste notes. That's going to have to be reconciled. Do we have the LLMs add "Cranial Nerves II-X intact" even though the physician did not actually assess that? The physician would have documented that... No? But then you open up the physician to liability, which is a no go for adopting software.
Building a SaaS MVP that's 80% of the way there? Sure. But medicine is not an MVP you cram into a pitch deck for a VC. 80% of the way there does not cut it here, especially if we're talking about consumer facing applications. Remember, the average American reads at a 6th grade reading level. Pause and let that sink in. You're probably surrounded by college educated people like yourself. It was a big shock when I started seeing patients, even though I am the first in my family to go to college. Any consumer-facing health AI tool needs to be bulletproof!!
Big Tech will not deliver us a healthcare utopia. Do not buy into their propaganda. They are leveraging post-pandemic increases in mistrust towards physicians as a springboard for half-baked solutions. Want to make $$$ doing the same thing? Do it in a different industry.
Claude just added Health Connect integration for Android.
Meanwhile Copilot launched a full bot for it:
"Dos and don’ts of medical AI
While AI is a useful tool that can help you understand medical information, it’s important to clarify what it’s designed to do (and what it isn’t).
Dos:
Use AI as a reliable guide for finding doctors and understanding care options.
Let AI act as an always available medical assistant that explains information clearly.
Use AI as a transparent, unbiased source of clinically validated health content.
Don’ts:
Don’t use AI for medical diagnosis. If you’re concerned you may have a medical issue, you should seek the help of a medical professional.
Don’t replace your doctor or primary care provider with an “AI doctor”. AI isn’t a doctor. You should always consult a professional before making any medical decisions.
This clarity is what makes Copilot safe"
https://www.microsoft.com/en-us/microsoft-copilot/for-indivi...
If an app makes a diagnosis or a recommendation based on health data, that's Software as a Medical Device (SaMD) and it opens up a world of liability.
https://www.fda.gov/medical-devices/digital-health-center-ex...
How do you suggest to deal with Gemini? Extremely useful to understand whether something is worrying or not. Whether we like it or not, it’s a main participant to the discussion.
Ideally, hold Google liable until their AI doesn’t confabulate medical advice.
Realistically, sign a EULA waiving your rights because their AI confabulates medical advice
Apparently we should hire the Guardian to evaluate LLM output accuracy?
Why are these products being put out there for these kinds of things with no attempt to quantify accuracy?
In many areas AI has become this toy that we use because it looks real enough.
It sometimes works for some things in math and science because we test its output, but overall you don't go to Gemini and it says "there's a 80% chance this is correct". At least then you could evaluate that claim.
There's a kind of task LLMs aren't well suited to because there's no intrinsic empirical verifiability, for lack of a better way of putting it.
Because $$$
Because privatized $$$ and public !!!
> How do you suggest to deal with Gemini?
Don't. I do not ask my mechanic for medical advice, why would I ask a random output machine?
This "random output machine" is already in large use in medicine so why exactly not? Should I trust the young doctor fresh out of the Uni more by default or should I take advises from both of them with a grain of salt? I had failures and successes with both of them but lately I found Gemini to be extremely good at what it does.
The "well we already have a bunch of people doing this and it would be difficult to introduce guardrails that are consistently effective so fuck it we ball" is one of the most toxic belief systems in the tech industry.
> This "random output machine" is already in large use in medicine
By doctors. It's like handling dangerous chemicals. If you know what you're doing you get some good results, otherwise you just melt your face off.
> Should I trust the young doctor fresh out of the Uni
You trust the process that got the doctor there. The knowledge they absorbed, the checks they passed. The doctor doesn't operate in a vacuum, there's a structure in place to validate critical decisions. Anyway you won't blindly trust one young doctor, if it's important you get a second opinion from another qualified doctor.
In the fields I know a lot about, LLMs fail spectacularly so, so often. Having that experience and knowing how badly they fail, I have no reason to trust them in any critical field where I cannot personally verify the output. A medical AI could enhance a trained doctor, or give false confidence to an inexperienced one, but on its own it's just dangerous.
> This "random output machine" is already in large use in medicine so why exactly not?
Where does "large use" of LLMs in medicine exist? I'd like to stay far away from those places.
I hope you're not referring to machine learning in general, as there are worlds of differences between LLMs and other "classical" ML use cases.
https://executiveeducation.mayo.edu/products/foundations-of-...
Instead of asking me to spend $150 and 4 hours, could you maybe just share the insights you gained from this course?
No, I'm not asking you spend $150, I'm providing you the evidence your looking for. Mayo Clinic, probably one of the most prominent private clinics in the US, is using transformers in their workflow, and there's many other similar links you could find online, but you choose to remain ignorant. Congratulations
The existence of a course on this topic is NOT evidence of "large use". The contents of the course might contain such evidence, or they might contain evidence that LLM use is practically non-existent at this point (the flowery language used to describe the course is used for almost any course tangentially related to new technology in the business context, so that's not evidence either).
But your focus on the existence of this course as your only piece of evidence is evidence enough for me.
Focus? You asked me for an evidence. I provided you with the one. And with the one which has a big weight on it. If that's the focus you're looking for then sure. Take it as you will, I am not here to convince anyone in anything. Have a look in the past to see how Transformers have solved the long standing problems nobody believed they are tractable up to that point.
An online course, even if offered by a reputable medical institution, hardly backs your argument.
LLM is just a tool. How the tool is used is also an important question. People vibe code these days, sometimes without proper review, but do you want them to vibe code a nuclear reactor controller without reviewing the code?
In principle we can just let anyone use LLM for medical advice provided that they should know LLMs are not reliable. But LLMs are engineered to sound reliable, and people often just believe its output. And cases showed that this can have severe consequences...
There's a difference between a doctor (an expert in their field) using AI (specialising in medicine) and you (a lay person) using it to diagnose and treat yourself. In the US, it takes at least 10 years of studying (and interning) to become a doctor.
Even so, it's rather common for doctors to not be albe to diagonise correctly. It's a guessing game for them too. I don't know so much about US but it's a real problem in large parts of the world. As the comment stated, I would take anything a doctor says with a pinch of salt. Particularly so when the problem is not obvious.
These things are not equivalent.
This is really not that far off from the argument that "well, people make mistakes a lot, too, so really, LLMs are just like people, and they're probably conscious too!"
Yes, doctors make mistakes. Yes, some doctors make a lot of mistakes. Yes, some patients get misdiagnosed a bunch (because they have something unusual, or because they are a member of a group—like women, people of color, overweight people, or some combination—that American doctors have a tendency to disbelieve).
None of that means that it's a good idea to replace those human doctors with LLMs that can make up brand-new diseases that don't exist occasionally.
It takes 10 years of hard work to become a profound engineer too yet it doesn't prohibit us missing the things. That argument cannot hold. AI is already wide-spread in medical treatment.
An engineer is not a doctor, nor a doctor an engineer. Yes, AI is being used in medicines - as a tool for the professional - and that's the right use for it. Helping a radiologist read an X-Ray, MRI scan or CT Scan, helping a doctor create an effective treatment plan, warning a pharmacologist about unsafe combinations (dangerous drug interactions) when different medications are prescribed etc are all areas where an AI can make the job of a professional easier and better, and also help create better AI.
And where did I claim otherwise? You're not disagreeing with me but only reinforcing my point
When a doctor gets it wrong they end up in a courtroom, lose their job and the respect of their peers.
Nobody at Google gives a flying fuck.
Not really, these are exceptionaly cases. For most of misdiagnoses or failure to diagnose at all, nothing happens to the doctor.
Why stop at AI? By that same logic, we should ban non-doctors from being allowed to Google anything medical.
Nobody can (and should) stop you from learning and educating yourself. It however doesn't mean just because you can use Google or use AI, you think you can become a doctor:
- Bihar teen dies after ‘fake doctor’ conducts surgery using YouTube tutorial: Report - https://www.hindustantimes.com/india-news/bihar-teen-dies-af...
- Surgery performed while watching YouTube video leaves woman dead - https://www.tribuneindia.com/news/uttar-pradesh/surgery-perf...
- Woman dies after quack delivers her baby while watching YouTube videos - https://www.thehindu.com/news/national/bihar/in-bihar-woman-...
Educating a user about their illness and treatment is a legitimate use case for AI, but acting on its advise to treat yourself or self-medicate would be plain stupidity. (Thankfully, self-medicating isn't as easy because most medication require a prescription. However, so called "alternate" medicines are often a grey area, even with regulations (for example, in India).
- The AI that are mostly in use in medicine are not LLMs
- Yes. All doctors advice should be taken cautiously, and every doctor recommends you get a second opinion for that exact reason.
> How do you suggest to deal with Gemini?
With robust fines based on % revenue whenever it breaks the law, would be my preference. I'm nit here to attempt solutions to Google's self-inflicted business-model challenges.
If it's giving out medical advice without a license, it should be banned from giving medical advice and the parent company fined or forced to retire it.
No.
"Whether we like it or not" is LLM inevitabilism.
https://news.ycombinator.com/item?id=44567857
Yes.
Counterpoint: LLMs are inevitable.Can't put that genie back in the bottle, no matter how much the powers-that-be may wish. Such is the nature of (technological) genies.
The only way to 'stop' LLMs is to invent something better.
depends if the cost of training can be recouped (with profit) from the proceeds of usage. Plenty of inventions prove themselves to be uneconomic.
Thought terminating cliche.
As a certified electrical engineer, the amount of times googles LLM suggested a thing that would have at minimum started a fire is staggering.
I have the capacity to know when it is wrong, but I teach this at university level. What worries me, are the people who are on the starting end of the Dunning-Kruger curve and needed that wrong advice to start "fixing" the spaces where this might become a danger to human life.
No information is superior to wrong information presented in a convincing way.
Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
A few months ago in a comment here on HN I speculated about the reason an old law might have been written the way it was, instead of more generally. If it had been written without the seemingly arbitrary restrictions it included there would have been no need for the new law that the thread was about.
A couple hours later I decided to ask an LLM if it could tell me. It quickly answered, giving the same reason that I had guessed in my HN comment.
I then clicked the two links it cited as sources. One was completely irrelevant. The other was a link to my HN comment.
Google after all has always been regurgitating what others say, true or false.
Lol thats funny. strange times
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me?
I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone must have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had.
Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
Yeah, this is by far the biggest value I've gotten from LLMs - just pointing me to the area of literature neither me nor any of my friends have heard of, but which have spent a decade running about the problems we're running into.
In this case, all that matters is that the outputs aren't complete hallucination. Once you know the magic jargon, everything opens up easily with traditional search.
The issue is when the output is good sounding but complete fantasy.
I’ve had it ‘dream’ up entire fake products, APIs, and even libraries before.
And that becomes obvious when you go to look for it. These work best in situations where false positives have low cost/impact, but true positives are easily verifiable and have high impact. in other words, problems in epistemological p-space. :)
I had a similar thing happen to me just today. A friend of mine had finished a book in a series. I have read the series but it was almost 10 years ago, and I needed a refresher with spoilers, so I went looking.
Well, some redditor had posted a comparison of a much later book in the series, and drawn all sorts of parallels and foreshadowing and references between this quite early book I was looking for and the much later one. It was an interesting post so it had been very popular.
The AI summary completely confused the two books because of this single reddit post, so the summary I got was hopelessly poisoned with plot points and characters that wouldn't show up until nearly the conclusion. It simply couldn't tell which book was which. It wasn't quite as ridiculous as having, say, Anakin Skywalker face Kylo Ren in a lightsaber duel, but it was definitely along those same lines of confusion.
Fortunately, I finished the later book recently enough to remember it, but it was like reading a fever dream.
Yeah, it happened recently for a kubernetes resource. I was searching for how to do something, and Google AI helpfully showed me a kubernetes resource that was exactly what I needed, and was designed to work exactly how I needed it.
Sadly, the resource didn't actually exist. It would have been perfect if it did, though!
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
It's a common problem.
At some point in time when asked how many Kurdish people live in Poland, Google's AI would say that several million, which was true, but only in a fantasy world conjured by a certain LARP group, who put a wiki on fandom.com.
Or you can take the alternative approach, where Microsoft's own "Merl" support agent says it knows anything to do with Minecraft, and then replies to basically any gameplay question with "I don't know that".
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it.
Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for.
I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
You can cure Google's AI (but not bad results) with &udm=14
https://google.com/search?q=parkas&udm=14
I've had the exact same issue with questions about EU5, but it mixed up EU4 and EU5 mechanics, making the answer just terrible.
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them... https://deadstack.net/cluster/google-removes-ai-overviews-fo...
Why does healthcare "need disruption"?
Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
When I look at the US, the symptom -> diagnosis hypothesis is not anywhere near the most expensive bit. If you have a medical issue and AI works effectively for this then it saves you maybe one trip to your GP. Your insurance probably still requires your GP to provide a referral to a specialist. If insurance companies allow for AI to be used in place of a referral then you save this trip. But you still need all of the stuff to confirm a diagnosis. And you still need all of the treatment.
If you don't have a medical issue and an AI system tells you this then you save yourself a trip to a specialist and the associated diagnostic tests. Again, this saves a bit of money but is nowhere near the bulk of medical expenses. And it has to be able to do this without any diagnostic testing, just based off of your reported symptoms.
Even if AI diagnosis works flawlessly we save a bit of money but absolutely do not revolutionize the cost of the industry.
Salaries for healthcare workers make up only a small portion of expenditures. You do not want to avoid a trip to your GP for an AI system.
It'll be great at first while in development. But when profits need to be generated, seeing a specialist will get harder. There will be less wiggle room. I predict we will see more GP utilization.
Right. But I'm saying that even in the best case scenario this disruption doesn't shift healthcare costs dramatically.
I mean, if we're talking Christensenian disruption, that happens in neglected markets rather than currently dysfunctional ones. There's no shortage of actors wronging money out of health care so there's not a disruptable space per se.
> wronging
Not sure if this is a typo (of wringing) or a pun, but it's apt either way.
Society pretends that human doctors are better than they really are, and AI is worse than it really is.
It's the self-driving cars debate all over again.
It's really just not there yet. I've been in medical school for >3 years now and have been using the latest models with good prompting. They have gotten much better, but I still see misses that my classmates would easily catch. This is not acceptable in healthcare. It's certainly not getting 100% on all my assignments, which are a step below the complexity of real-world clinical practice.
Before medical school, I was not so sure of the quality of your average doc. Now having spent a year in clinical practice across various settings, I am extremely reassured. I can say with certainty that a US trained doctor is miles ahead of AI right now. The system sucks really bad though and forces physicians to churn patients, giving the impression that physicians don't pay attention/don't care/etc.
because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
Physician compensation is around 9% of healthcare spending. The number of non-physician providers (NPs, PAs and specialists like physical therapists and podiatrists) has also exploded over the last 20 years. We have far more overall providers per capita than we did 20 years ago.
Lack of providers isn’t what’s driving up costs.
I don't think physician compensation per se is a good metric for capturing the effect of lack of providers, because some of the increased costs are due to the bottlenecks in the services per se, in terms of procedure costs and types of procedures offered. I also don't think the number of providers per se under the current regime, without deregulation or reregulation of practice boundaries, is representative of what would happen if there were changes in those boundaries. Adding more optometrists 5 years ago isn't the same as changing what they're allowed to do. It also doesn't address what cost increases would have been without an increase in the number of providers.
9% might also seem pretty big to me if it's out of all spending and doesn't include other provider compensation? What if overall healthcare costs went down, but physician compensation stayed the same? Would that then be a problem because it was an increased proportion of the total costs — fat left to be trimmed, so to speak?
There are many problems that don't have anything to do with providers per se, but I also don't think you can glean much by extrapolating to more of the same, especially compensation per se.
If lack of physicians is leading to an increase in costs, you’d expect to see physicians capturing a large part of that increase. There are situations where that wouldn’t hold, but it requires moving away from the simplest explanation that fits the data.
We don’t have to extrapolate from physician compensation though. We know that providers per capita have increased, but costs have continued to skyrocket. Therefore a lack of providers is not the immediate cause of the increase.
In addition to increasing the number of providers, the scope of practice for non-physician providers has almost universally increased.
All of this doesn’t prove that increasing the number of physicians wouldn’t lower costs some amount, but it does show that the increases over the last 20-30 years requires some other explanation.
9% is nothing when you consider that these people provide the value. With no physicians, there is no healthcare.
Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
I sympthatise with the argument. We should test it against real world data.
Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
Dental and vanity surgeries aren't happening in a vacuum. There are baseline costs eg. anesthesia, recovery medications, medical machinery etc which are all bloated due to the rest of industry not being under price pressure (rising tide lift all boats)
It's similar to how AI data center buildout race is raising the prices for consumer electronics in 2026 and beyond. The suppliers have no incentive to sell lower cost products to tiny niche
How does Lasik manage to drop in real price, then?
I just looked it up, and apparently health care costs for pets has gone up in price even faster than for humans.
Pets typically don't have medical insurance, and any insurance that does exist there has a radically different regulatory regime than for humans.
Since 1980 for the US:
CPI has gone up by 3.16% on average per year (x4.17 in total). Human healthcare costs by 4.9% per year (x8.96 in total). And pet healthcare costs by 6.49% (or x17.87 in total).
I suspect this has something to do with it: https://www.theatlantic.com/ideas/archive/2024/04/vet-privat...
That could at most explain the last few years of rising vet prices.
Technological advancement seems like a pretty big culprit here? An MRI wasn't available at any price in 1982 so the fact that it's available now makes it look like "imaging prices have gone up" even though the imaging that was available in 1982 is cheaper now than it was then.
The uninsured medical market is actually quite efficient. e.g. lasik surgery can actually be done for very reasonable rates with full price transparency.
Lasik has gone down in real price over the years, yes.
But dental and vanity cosmetic surgery have gone up by that metric. Dental is less covered by insurance for most people. Vanity cosmetic insurance is covered for almost no one.
Vet care for pets has gone up a lot more than healthcare for humans.
This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
The real source of high medical costs is the entity that sets the hospital bill in the first place.
The explanation is much simpler than people want to admit, but emotionally uncomfortable: doctors and hospitals are paid more than the free market would otherwise justify. We hesitate to say this because they save lives, and we instinctively conflate moral worth with economic compensation. But markets don’t work that way.
Economics does not reward people based on what they “deserve.” It rewards scarcity. And physician labor is artificially scarce.
The supply of doctors is deliberately constrained. We are not operating in a free market here. Entry into the profession is made far more restrictive than is strictly necessary, not purely for safety, but to protect incumbents. This is classic supply-side restriction behavior, bordering on cartel dynamics.
See, for example: https://petrieflom.law.harvard.edu/2022/03/15/ama-scope-of-p...
We see similar behavior in law, but medicine is more insidious. Because medical practice genuinely requires guardrails to prevent harm and quackery, credentialing is non-negotiable. That necessity makes it uniquely easy to smuggle in protectionism under the banner of “safety.”
The result is predictable: restricted supply, elevated wages, and persistently high medical costs. The problem isn’t mysterious, and it isn’t insurance companies. It’s a supply bottleneck created and defended by the profession itself.
Insurance companies aren't innocent angels in this whole scenario either. When the hospital bill fucks them over they don't even blink twice when they turn around and fuck over the patient to bail themselves out. But make no mistake, insurance is the side effect, the profession itself is the core problem.
> This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
They absolutely do not.
They have their profit levels capped at 15% by law and regulation. That means if the insurer wants more absolute dollars of profit, prices must go up.
It also means that if they push prices down they necessarily have less funding to administer those plans, even if the needs are the same (same number of belly buttons, same patient demographics and state of health).
As you note there's also other variables, but this claim: "Insurance companies are structurally incentivized to minimize payouts across the board" is absolutely and categorically not so.
Physician reimbursement is only ~9% of national healthcare expenditures.
I tell you this with certainty as a 3rd year medical student: If physician wages go down and tuition stays as is, no one will do this. Intrinsic motivation to help people evaporates as soon as you see how enshittified healthcare in the US has become.
I do agree that medical school is far too restrictive to get into (For MD schools at least). However, if you want to make medical school easier to get into: Where will all those students rotate at for their clinical years? There aren't enough spots in hospitals to jam students in.
Stop taking aim at the people that sacrifice so much to help you. Take aim at the real drivers of healthcare expenditures: administrative bloat.
>Where will all those students rotate for their clinical years? There aren’t enough hospital slots.
This is a policy fiction. Residency slots are capped by federal law, not by hospital capacity. The Balanced Budget Act of 1997 froze Medicare-funded residency positions, and despite modest expansions decades later, the cap remains largely intact. Teaching hospitals routinely report excess clinical volume relative to trainee supply. The bottleneck is artificial and regulatory, not logistical.
>Stop taking aim at people who sacrifice so much to help you. The real cost driver is administrative bloat.
This framing collapses under scrutiny. Administrative bloat is real and well-documented, but pretending physician incentives are irrelevant requires willful blindness. Numerous studies show that U.S. physicians earn multiples of their OECD peers while delivering no commensurate advantage in outcomes. Many doctors are motivated by altruism, but many are also motivated by status, income, and professional gatekeeping—normal human incentives in a high-prestige, high-pay profession.
Further, high patient throughput is not an accident. Fee-for-service reimbursement structurally rewards volume over care quality. Seeing 20–30 patients a day is not a moral failure of individual doctors, but it does predictably lead to burnout, emotional detachment, and assembly-line medicine. Incentives shape behavior. Ignoring that is not compassion, it’s denial.
>Physician reimbursement is only ~9% of national healthcare spending.
That statistic is repeatedly used as a rhetorical shield, and it shouldn’t be. Cost systems do not fail because of a single oversized line item; they fail because multiple protected constituencies simultaneously extract rents while deflecting blame. Administrative overhead, defensive medicine, pharmaceutical pricing, hospital consolidation, reimbursement incentives, and physician compensation are jointly optimized for revenue, not outcomes.
Nine percent of a multi-trillion-dollar system is not trivial. More importantly, physician compensation is not isolated—it drives downstream costs through referral patterns, test ordering, procedure rates, and resistance to scope-of-practice reform. Treating physicians as a sacred class exempt from economic critique is precisely how you end up with a system that is unaffordable, unaccountable, and structurally resistant to reform.
If the argument is “9% is too small to question,” then by that logic no component is ever large enough to examine in isolation, which is how dysfunctional systems persist indefinitely. Real reform requires abandoning moralized narratives and admitting the obvious: healthcare costs are the product of aligned incentives across many actors, and physicians are not magically outside that system simply because the story is uncomfortable.
It's inefficient and not living to its potential
And "disruption" (a pretty ill-defined term) is the solution to that?
The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here.
And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
Maybe for R&D but your outsized costs are also due to your liability environment and direct incentives against preventative healthcare.
We'd rather have medical bankruptcy than a functional system.
That's because it's incredibly corrupted, not because it needs disruption. Unless the disruption comes in the form of jail time.
The inefficiency is the buying of yachts for billionaires.
Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
If we could get healthcare to that level, it would be great.
For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.
> Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
Google searches cost many billions of dollars: your confusion is because the customer isn’t the person searching but the advertisers paying to influence them. Healthcare can’t work like that not just because the real costs are both much higher and resistant to economies of scale but, critically, there aren’t people with deep pockets lining up to pay for you to be healthy. That’s why every other developed country sees better results for less money: keeping people healthy is a social good, and political forces work for that better than raw economic incentives.
Wal-Mart and Amazon have reduced wages for employees and the quality of purchased goods more than they have improved prices for consumers.
How do we know that?
And why do customers come back to shop there?
We know that from observing evidence such as how much the government pays out in welfare to Wal-Mart employees.
Customers continue shopping there because human beings are typically incapable of accepting a short-term loss (higher price) for a long-term gain (product lasts more than three uses).
> We know that from observing evidence such as how much the government pays out in welfare to Wal-Mart employees.
That's a weird metric. If tomorrow Wal-Mart laid off all employees and replaced them with robots, they would surely be worse off, but by your metric Wal-Mart would look less evil?
> Customers continue shopping there because human beings are typically incapable of accepting a short-term loss (higher price) for a long-term gain (product lasts more than three uses).
Groceries typically only last one use.
I mean, I could tell you are disingenuous from the get-go, so it is not surprising you would take an off-the-cuff metric which is accurate right now and invent a strawman scenario where I might continue to use it beyond a point where it makes sense.
Likewise, I would not use my flippant 3 times metric regarding durability to cover the quality of produce.
Well, feel free to propose a different metric for seeing how Wal-Mart is evil. And instead of Wal-Mart firing all employees, we can have a look at the more plausible scenario of Aldi vs Wal-Mart. Aldi runs a different labour model: their shifts are much more intense but pay more. Their model can't and doesn't utilise the kind of older and less fit people that Wal-Mart employs.
You have to look at the counterfactual of what these people would do, if Wal-Mart weren't around. You seem to implicitly assume that they'd be getting higher paying jobs somewhere else (so they wouldn't have to rely on welfare)? If so, what's stopping those people from switching to these better jobs right now, even while Wal-Mart is still around?
And sure, let's disregard how many times you can eat your groceries. That was a cheap shot. However I think quality vs price trade-off is something customers have to make for themselves anyway. Who am I to judge their choices?
And Google search, a service on the level of a public utility, has been degrading noticeably for years in the face of shareholders demanding more and more returns.
How is Google Search a public utility?
Comparing something to a public utility is not me saying it's literally a public utility. Google runs a monopolistic service that is essential to a lot of our public life, in a segment that has high cost of entry and infrastructure cost. They make the service worse to make more money. It should be a regulated utility like electricity or railroads, we should have a public alternative like the post office is to UPS, or it should be nationalized. The situation gets more dire when you consider their browser monopoly.
Other search engines exist. Bing is right there, and Microsoft is more than willing to eat the high cost of entry and infrastructure cost.
> It should be a regulated utility like electricity or railroads, we should have a public alternative like the post office is to UPS, or it should be nationalized.
I agree that electricity and railroads should be regulated like Google Search.
It's really weird that snail mail in the US is a government monopoly. When even social democratic Germany managed to privatise them.
> The situation gets more dire when you consider their browser monopoly.
Don't a lot of people in the US use iPhones? They don't ship with Chrome as the default browser, do they?
(And yes, Safari is built on top of the same open source engine as Chrome. But you can hardly call using the same open source project a 'monopoly'. Literally anyone can fork it.)
There's also plenty of other browsers available.
The existence of few competitors is not proof that monopolistic power doesn't exist and isn't being leveraged. Saying Google isn't monopolistic is being willfully wrong. You're more wrong when we look at the browser market, and Google has lost anti-trust suits on this very topic in the past couple years.
A public mail service is required by our constitution. It's cheaper than the private options and often the only option for many rural areas. It's not a monopoly.
> A public mail service is required by our constitution.
Where does it say so in your constitution? All I can find is the postal clause which Wikipedia summarises as follows, but whose full text isn't much longer:
> Article I, Section 8, Clause 7, of the United States Constitution, the Postal Clause, authorizes the establishment of "post offices and post roads"[1] by the country's legislature, the Congress.
https://en.wikipedia.org/wiki/Postal_Clause
The Postal Clause certainly allows the government to run a public postal service, but I don't see how the constitution _requires_ it. It doesn't even require the federal government to regulate postal services, it merely allows it.
Perhaps I missed something?
> It's cheaper than the private options and often the only option for many rural areas.
If you want to subsidise rural areas, I would suggest to do so openly, transparently and from general taxation. At least general taxation is progressive etc. Instead of just making urban folks pay more for their mail, whether they be rich or poor.
I would also suggest only subsidising poor rural areas. Rich rural areas don't need our help.
> It's not a monopoly.
Compare and contrast what USPS has to say https://about.usps.com/universal-postal-service/universal-se...
IDK, the owners of retail clothing chains buy yachts and yet that sector is jaw-droppingly efficient at delivering clothes to people. Executives can be annoying tools but I don't think their pay is the problem.
Shame those same owners have plenty of money for buying yachts and not enough money to pay their sweat-shop employees a living wage or to provide decent work conditions. Executives don't "earn" huge paychecks, they merely exploit others by figuring out a way to not pay them their worth.
It's inefficient and not living to its potential
Yeah, because we saw what a great job the tech bros did making government more efficient.
Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
My fiance was in hospital recently for a fairly common disease. She arrived at 2200 Wednesday night and was discharged 1000 Saturday morning.
Her bill before "insurance negotiated prices" was $59,000. Effectively $1,000/hr, 24/7.
But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.
> large insurance pools that must span age groups and risk groups.
What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.
> Mandate at least decent minimal coverage standards
I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)
Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.
There are a lot of not-for-profit insurance companies and they aren't noticably cheaper, though I'm not in HR and they may well be cheaper for the employer.
Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?
Profit isn't even a big part of the overall revenue.
> Mandate at least decent minimal coverage standards
I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.
> and large insurance pools that must span age groups and risk groups.
Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
> Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
You are mixing things up.
There's insurance which allows you to convert an uncertain danger into a known payment. And then there's welfare and redistribution.
By all means, please run some means testing and give the poor and sick or disabled extra money. Or even just outright pay their insurance premiums.
But please finance that from general taxation, which is already progressive. Instead of effectively slapping an arbitrary tax on healthy people, whether they be rich or poor. And please don't give rich people extra stealth welfare, just because they are in less than ideal health, either.
Just charge people insurance premiums in line with their expected health outcomes, and help poor people with the premiums using funds from general taxation. (Where poor here means: take their income and make an adjustment for disability etc.)
We _want_ the guy who loses 5kg and gives up smoking to get lower insurance premiums. That's how you set incentives right.
> The more diverse the pool, the lower the risk.
No. The diversification comes from the insurance company running lots of uncorrelated contracts at the same time and having a big balance sheets. For that, it doesn't matter whether it's a pool of similar insurance contracts, or whether they have bets on your insurance contract, and on the price of rice in China, and playing the bookie on some sports outcomes etc. In fact, the more diversified they are, the better (in principle).
But that diversification is completely independent of the pricing of your individual insurance contract.
Have a look at Warren Buffett's 'March Madness' challenge, where he famously challenges people to predict all 67 outcomes of some basketball games to win a billion dollar. Warren Buffet ain't no fool: he doesn't need a pool, he can price the risk of someone winning this one off challenge.
More generally, have a look at Prize indemnity insurance https://en.wikipedia.org/wiki/Prize_indemnity_insurance which helps insure many one-off events.
> There's insurance which allows you to convert an uncertain danger into a known payment. And then there's welfare and redistribution.
Well, there I entirely agree with you - health insurance as it exists in the US now is "insurance in name only".
This is going to pretty rapidly devolve into cheap for healthy and insanely expensive for those that aren’t. Genetic propensities will be a lifelong financial burden. Cancer patients will get priced out and die.
If you want to subsidise these unfortunate people, please just do so openly. Don't be all sneaky, sneaky about it by banning accurate risk assessments.
In any case, what you are saying is only true, if you buy your health insurance second to second on the spot market.
Insurance companies are more than happy to enter long running contracts, where you both agree today on (the algorithm for) the premiums for the next twenty years or even until the rest of your life. That's pretty common with life insurance and disability insurance already.
The above already exists, but if you allow some speculation: you could even envision people buying insurance for their kids before conceiving them. That way you don't have to worry about pre-existing conditions.
(Well, if the parents already have heritable conditions that would make the kids more likely to have expensive medical problems, those would push up their premiums. But then: perhaps these people should think twice about burdening a potential kid with these issues.
Compare how in Cyprus where sickle cell anemia is prevalent, even the Catholic church demand you get screened, before they'll marry you.)
If you really want specialised in-kind welfare, you can get people a voucher for the catastrophic version of 'unconceived baby insurance'.
Basically, you can buy insurance against insurance premiums being expensive.
Disruption, yes, in the sense that the current system needs to be overhauled. But this is a space that's frequented by the SV and VC space and "disruption" has very different connotations, usually in the realm of thought that suggests some SV-brained solution to an existing problem. In some edge cases like Uber/Lyft, this upending of an existing market can yield substantial positive externalities for users. Other "heavy industry" adjacent sectors, not so much. Healthcare and aviation, not so much.
Even SpaceX's vaunted "disruption" is just clever resource allocation; despite their iterative approach to building rockets being truly novel they're not market disruptors in the same way SV usually talks about them. And their approach has some very obvious flaws relative to more traditional companies like BO, which as of now has a lower failure-to-success ratio.
I don't think you'll find many providers clamoring for an AI-assisted app that hallucinates nonexistent diseases, there are plenty of those already out there that draw the ire of many physicians. Where the industry needs to innovate is in the insurance space, which is responsible for the majority of costs, and the captive market and cartel behavior thereof means that this is a policy and government issue, not something that can be solved with rote Silicon Valley style startup-initiated disruption; that I would predict would quickly turn into dysfunction and eventual failure.
Enshittification has done a lot of damage to the concept of "disrupting" markets. It's DOA in risk-averse fields.
Why would "disruption" make it better rather than worse?
The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
Profit is a small part of overall revenue.
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
What medication is this? It's always given me good info.
I'm not sure why you would think I would blurt that out on the internet.
And are you sure it's giving you good info? "AI" is famously subject to hallucinations, so you may not be getting the "good info" you think you're getting. Be careful with "AI", it's not an all-seeing-all-knowing infallible oracle.
I think you would blurt that out on the internet because nobody here has any idea who you are
I do not rely on the AI summaries from Google for health related inquiries. It has proven wrong too often.
-Med student
I do not rely on the conclusions of doctors because they have proven wrong too many times.
- guy with narcolepsy who was incorrectly diagnosed by a dozen or so doctors
s/hallucinated/fabricated/, please.
arguably: incorrectly guessed*
in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be
"confabulation", though, seems perfect:
“Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.”
https://en.wikipedia.org/wiki/Confabulation
* insofar as “guess” conveys an attempt to be probably in the zone
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
What do you use instead… that doesn’t piggyback off of google search?
The AI summaries clearly aren’t bad. I’m not sure what kind of weird shit you search for that you consider the summaries bad. I find them helpful and click through to the cited sources.
...and the cited source is AI generated video(s). There are summaries that say exactly the opposite of the correct result.
> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added.
Notice how little this sentence says about whether anything is any good.
I can't speak to Health summaries, but the Weather summary on Pixel phones is not good.
Like you could have a few days of -3C, for today it goes up to +5C, and the "AI Weather report" tells you it's going to be a chilly day or something.
I never saw this feature provide any useful information whatsoever.
No sane, in the loop person would believe Google reviews anything, at least not with a human. I bet they have another AI reviewing the answers, and the only time a human enters the loop is when this second AI needs to be restarted.
Tangent, but some people I know have been downloading their genomes from 23andme and asking Gemini via Antigravity to analyze it. "If you don't die of heart disease by 50, you'll probably live to be 100."
I wonder how accurate it is.
23&Me data is simply not accurate enough to make reasonable predictions about health outcomes.
I have a whole genome and nothing Google has built has been able to do anything useful with it, medically speaking. I could use DeepVariant to re-map all the raw reads, it would only slightly increase the accuracy of the estimate of my genome sequence. When I met with genetic counselors, they analyzed my genome and told me I had no known markers for any disease (and they also told me they Google all the unique variants that show up in the report).
(for what it's worth, I literally went to work at Google to improve their medical/health/genomics research, and after working on it a few years I concluded that the entire genomics field is about 90% fantasy. If you want actionable data, there are a small number of well-regulated tests that can help in a limited set of circumstances, but those aren't whole genome tests).
As accurate as our knowledge of genetics, which is not very outside of the identified set of pathological genes associated with hereditary disorders.
Your genome is very complex and we don’t have a model of how every gene interacts with every other and how they’re affected by your environment. Geneticists are working on it, but it’s not here yet.
And remember that 23andMe, Ancestry, and most other services only sequence around 1% of your genome.
I'd guess it's much less accurate than that.
Part of genetics is pattern matching, and last time I checked I still can't find a model that can correctly solve hard Sudokus (well, assuming you don't pick a coding model that writes a Sudoku solver.. maybe some of them are trying to do genetics by doing correct algorithms), a trivial job if you write a program that is designed to do it.
Are you asking for you in particular? It's certainly not accurate in general that anyone that made it to 50 is likely to live to 100.
One I heard was if you make it to 80 you have a 50% chance to make it to 90. If you make it to 90 you have a 50% chance to make it to 95. From 95 to 97.5 again 50% chance. That for the general population, in a 1st world country though, not any individual.
The assessment he got was "you have supergenerian genes, but also this one that's really sensitive to heart disease. If you can keep you plaque in check, you're almost invincible (to other popular causes of mortality)."
It cited whatever gene it said he had that made that so.
Our understanding of genomic data is in its infancy. These people are idiots who don't know what they don't know.
But only for some highly specific searches, when what it should be doing is checking if it's any sort of medical query and keeping the hell out of it because it can't guarantee reliability.
It's still baffling to me that the world's biggest search company has gone all-in on putting a known-unreliable summary at the top of its results.
This incessant, unchecked[1] peddling is what rids "AI" of the good name it could earn for the things it's good at.
But Alas, infinite growth or nothing is the name of the game now.
[1] Well, not entirely thanks to people investigating.
... at the same time, OpenAI launches their ChatGPT Health service: https://openai.com/index/introducing-chatgpt-health/, marketed as "a dedicated experience in ChatGPT designed for health and wellness."
So interesting to see the vastly different approaches to AI safety from all the frontier labs.
Why vastly different?
Aren't they both searching various online sources for relevant information and feeding that into the LLM?
Different levels of capabilities. The summary feature in google uses a quick and inaccurate AI model. Were it to be a heavier model, we wouldn’t have this problem.
We would still have this problem. The heavier models make mistakes at too high a rate vs. a physician. Especially on imaging data. Real world data and patient presentations often deviate from the textbooks they are trained on.
-Med student
That's a different class of problem. It will do just fine on text based queries spanning a few pages. Probably better than the average physician (average over all countries).
I do agree that LLM's are not there yet in the image part.
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy.
Also try "health benefits of circumcision"...
I agree with your point.
Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications.
I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
the problem isn't that search engines are polluted; that's well known. The problem is that people perceive these AI responses as something greater than a search query; they view it as an objective view point that was reasoned out by some sound logical method -- and anyone that understands the operation of LLMs knows that they don't really do that, except for some very specific edge examples.
It took being a meme for a literal year for them to remove this… more responsibility in frontier tech, I’m begging.
The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.
Ars rips of this original reporting, but makes it worse by leaving out the word "some" from the title.
‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk: https://www.theguardian.com/technology/2026/jan/11/google-ai...
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk"
Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"
How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?
Being gutted by DOGE and the Trump Administration under RFK Jr.
I'm telling you all this as a medical student that has used the latest and greatest models with proper prompting for the past 3 years in school:
There are a ton of misses. Especially on imaging. LLMs are not ready for consumer-facing health information yet. My guess is ~ 3-5 years. Right now, I see systems implementing note writing with LLMs, which is hit or miss (but will rapidly improve). Physicians want 1:1 customization. Have someone sit with them and talk through how they like their notes/set it up so the LLMs produce notes like that. Notes need to be customized at the physician level.
Also, the electronic health records any AI is trained on is loaded to the brim with borderline fraud/copy paste notes. That's going to have to be reconciled. Do we have the LLMs add "Cranial Nerves II-X intact" even though the physician did not actually assess that? The physician would have documented that... No? But then you open up the physician to liability, which is a no go for adopting software.
Building a SaaS MVP that's 80% of the way there? Sure. But medicine is not an MVP you cram into a pitch deck for a VC. 80% of the way there does not cut it here, especially if we're talking about consumer facing applications. Remember, the average American reads at a 6th grade reading level. Pause and let that sink in. You're probably surrounded by college educated people like yourself. It was a big shock when I started seeing patients, even though I am the first in my family to go to college. Any consumer-facing health AI tool needs to be bulletproof!!
Big Tech will not deliver us a healthcare utopia. Do not buy into their propaganda. They are leveraging post-pandemic increases in mistrust towards physicians as a springboard for half-baked solutions. Want to make $$$ doing the same thing? Do it in a different industry.
They AI summary is total garbage. Probably most broken feature I saw being released in a while.
huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol
Meanwhile ChatGPT health just 5 days ago https://openai.com/index/introducing-chatgpt-health/ because clearly they both don't give a fuck AND they are so desperate for money, any money.
Claude just added Health Connect integration for Android.
Meanwhile Copilot launched a full bot for it:
"Dos and don’ts of medical AI While AI is a useful tool that can help you understand medical information, it’s important to clarify what it’s designed to do (and what it isn’t).
Dos: Use AI as a reliable guide for finding doctors and understanding care options. Let AI act as an always available medical assistant that explains information clearly. Use AI as a transparent, unbiased source of clinically validated health content. Don’ts: Don’t use AI for medical diagnosis. If you’re concerned you may have a medical issue, you should seek the help of a medical professional. Don’t replace your doctor or primary care provider with an “AI doctor”. AI isn’t a doctor. You should always consult a professional before making any medical decisions. This clarity is what makes Copilot safe" https://www.microsoft.com/en-us/microsoft-copilot/for-indivi...
[dead]
[flagged]
What's that?
lol you must be fresh to the internet
chatGPT told me, I am the healthiest guy in the world, and I believe it
If it's any help women appreciate confidence; Yet, the article is not about chatgpt