> But let’s think about that 91% failure rate for a moment. When I bring this up in presentations, I invite the audience to consider what the auto industry would look like of 91% of new car designs proved unable to roll out of the factory, or if 91% of new airliner models were unable to leave the ground - and if you only found that out after spending all the R&D money to build them at full size and trying to fly them. No cutting-edge restaurant could survive if 91% of its innovative dishes proved inedible or outright poisonous. What other industries operate under these bizarre conditions?
This is bizarre, bordering on stupid. Frist of all, there likely is ~90% failure rate of prototypes; I feel that roughly matches my experience in engineering. Of course the design that makes it through the process, testing, refinement, and into mass production is not going to have a 90% failure rate, but that's a _finished product_, whereas clinical tests are just that -- tests. Finished cars are more analogous to individual pills coming out of the factory. And I'm not even sure the analogy would be very meaningful anyways, because we have different requirements for things of different impact and importance. A 10% manufacturing defect rate is fine in forks, but not for fire extinguishers.
It's not that much better in plant breeding: You'll find hundreds upon hundreds of supposedly better varieties tried, and 5 years later, you are lucky to get 1 good one that one would consider commercializing. And that's after years of testing changes in smaller settings. We just don't have models that are predictive enough without planting, and waiting for plants to grow takes time.
A large part of that failure rate is in phase 3. Your comparison with prototypes would be more like drug development before even phase 1. Phase 3 is enormously expensive, much more than the earlier parts. And what's even worse, you don't get to learn all that much from failures in drug development in many cases. You can't just fix the problem and try again, you essentially have to try a completely new molecule.
That's fair, phase 3 testing isn't an early prototype, it's a prototype with many years of effort behind it. But I still think it is _more_ like a prototype than it is like a product coming off an assembly line. Perhaps phase 3 is like the latest prototype from boom (the company trying to bring back super-sonic jets). Those prototypes represent 100s of engineering-years put together, and are not much like the prototypes you'd encounter in consumer product development. I still think 90% failure rate is believable for that. How many significant design iterations has Boom gone through so far? I don't really know, but I could believe it's been 10 significant iterations.
Yeah why is 10% success bad? What is the tradeoff between spent effort and missed cures if we try to tweak the success rate by killing prototypes earlier in the pipeline? While reading I was expecting a reasoned argument... that was stubbornly not coming around.
It's mostly that we have too many false findings pre-clinical trials, most drug targets validated in models that don't actually hold up in humans, so a huge share of the 90% failure is money and years spent testing candidates that were never going to work. That inflates the number of potential candidates that are likely wrong, due how we select them.
There are ways used right now that are working towards reducing those false findings by looking at actual humans, their biomarkers and whenever or not there's an associated molecule to the condition that we would like to pass onto others. It will not kill prototypes, instead it will discourage us from going through a prototype at all by going for better candidates instead.
The problem is that you model the filtering step as a monolithic step.
Things start with academic literature, and models (qualitative or even numerically quantitative, human comprehensible, or computationally predicted effects, ...). Then academic level testing occurs, resulting in putative results, obtained on animal models, cell / tissue cultures, ... before proceeding to human trials. The failure rates are for this final step. Imagine being in control of some pharma fund, there is a huge stream of putative drugs emerging in the literature, and only limited budget AND limited test-patient slots. On average people in this position succeed in selecting one that performs as predicted only ~10% of the time. Thats not a simple exploration-exploitation trade-off. With so many candidates, assuming proper pre-human experiments, one would expect much better results, and you'd from a financial perspective redirect focus towards those drugs with high confidence from prior forms of non-human testing. Yet we see failure rates 80-90%! From a purely financial perspective, there is a huge incentive to place more selection emphasis on confidence, but either its not happening or institutions (public or private) are systematically dropping the ball.
To make the engineering analogy with design methodology: before testing a new implementation, we have the luxury to preselect implementations depending on their unit tests, considerably improving the success rate for the higher level implementation. Yet for the analogy in drug selection we fail miserably.
I don't believe the failure is proper to the selection process. Its that the true fitness function is unavailable, if we had it would simply be a matter of performing gradient descent.
But obviously earthly biology does not come with a reference manual of all niches, and analytic objective differentiable fitness functions.
To a large extent it is in fact still the exploration-exploitation curve, but a meta level.
We can't bypass natural selection, we can't turbocharge natural selection (like the Nazi's tried), not only because it is evil, but because the fitness statistic is for all purposes and intents, emergent in nature.
The average reader here will be very familiar with the concept of premature optimization: don't start micro optimizing your code in assembler before functional correctness, first go for correctness, then reason about the hot paths from profiling and investigate and optimize from there.
Historically, hospitals and medicine long predate modern science and biology.
It predates the discovery of natural selection, it predates the measurement of selection phenomena on shortlived organisms.
Healthcare is high inertia never-to-seldom-recognize-earlier-mistakes domain.
The justification of healthcare is never supported by some axiomatized formally verifiable system of ethics, it is justified on the basis of associations and vague nebulous historically grown rules.
Socialized healthcare is a world-widely supported doctrine.
Don't wish unto others what you wouldn't wish onto yourself is another.
There is some nebulous concept of a right to healthcare, but a right to what: some nebulous default "healthy" state?
There is also a strong connection with egalitarianism, if someone gets sick from a flu we somehow believe it is desirable to help them overcome it, because we claim this flu could have afflicted any individual equally.
THe healthcare zooko's triangle looks like this, you can't have all 3 of the following, so once you have unconditionally selected one desideratum, you will be left torn by the decision between only one of the remaining desiderata:
1. egalitarian access to healthcare
2. working healthcare solutions towards "individual health", i.e. improving procreation rates statistically
3. maintained fitness of the collective human genome distribution
Assuming without compromise 1: egalitarian access to healthcare:
===========
To the extent a healthcare instrument (drugs, or tools like glasses) works (2), it undoes decreased genetic procreation rates, inducing a higher incidence rate in future generations, reducing the fitness of the human genome (3): we have not cured or treated this individual, we have traded innate health, fitness and quality of life of future generations for the convenience and comfort of individuals in the current generations.
To the extend we insist to maintain genetic fitness, it would require the healthcare instrument to be ineffective and thus not influence procreation statistics of fertile individuals. To the extent a healthcare instrument doesn't affect the procreation statistics of a fertile individual, the healthcare instrument didn't improve quality of life for this individual (and is effectively a quack measure). For example without the flu shot, an fertile individual might have met a potential mate, or been in the mood to mate with an already associated partner, but an individual without the flu shot might have felt too sick, or been to repulsive for a mate or partner during sickness. Natural selection first and foremost is about procreation probabilities and rates, and much rarer the individually stronger but collectively weaker signal of death.
For example: there is wide consensus that pre-modern tribal hunter/gatherer humans only had sub 5% incidence rates of poor vision requiring corrective glasses according to modern standards. The selective pressure on eyesight was so strong that even 1 or 2 generations of modern medicine, results in large majority of population requiring prescription glasses, the loss of a strong selective pressure quickly results in drift away from fitness.
Assuming without compromise 2. "individually effective" healthcare instruments:
=============
If we select to keep (1) egalitarian access then it will be at the expense of (3) maintained fitness in the humanities collective genome: the egalitarian access to the individually effective healthcare measure, will result in the loss of genetic utility, since the utility is supported and provided externally instead of innately.
If we select to maintain the collective genetic fitness of humanity (3), it can be attained while maintaining "individually effective healthcare" measures (drugs, crutches, pacemakers,...), but only if we relax (1) egalitarian access to healthcare: a modern nation state can admit migration or otherwise support the transfer of genetic material from regions with no or little healthcare, or from regions where such healthcare was only recently introduced. Steady state reliance on such a stream of wild-type humans, is basically bio-colonialism: it requires a region where humans face natural selection without help from healthcare, and their genes are used to improve or maintain fitness of modern nation states elsewhere.
Assuming without compromise (3) maintaining fitness of humanities collective genome:
If we insist on maintaining (1) egalitarian access; then it will be at the cost of (2) healthcare that objectively improves the quality of life and thus procreation statistics in the case of fertile individuals. So we could all enjoy egalitarian access to non-functional medicine.
If we insist on maintaining (2) individually effective healthcare measures, which improve quality of life and hence procreation statistics of fertile individuals, we have to sacrifice (1) egalitarian access: if a sufficient collection of humans is basically deprived of healthcare access, then yes we could maintain fitness by selecting their genetics for procreation.
It's a veritable zooko's triangle, and an absolute nightmare once comprehension sinks in.
If for every QAPR (quality adjusted procreation rate) were taken into account for healthcare interventions, they'd all go negative! whatever current "meritocratic assessment" of healthcare instruments is basically fraud from the perspective of genetics and natural selection, like those video's you see from parents climbing school walls in India to help their children cheat on some national level exams, and somehow this being normalized by society...
It is not a question of allegiance with or against modern healthcare, it is a question of internal consistency of what is known about natural selection, selection pressure, healthcare, etc. We only know how to trade with loss (but you don't need a degree in medicine to attain that skill).
Modern medicine stems from a premature optimization objective (and the original goals did not involve (1) egalitarian access -for nobility and their armies- nor did the original goals involve (3) maintaining genetic fitness for future generations -the statistics of natural selection and concepts like selection pressure were only poorly understood).
Socialized healthcare institutions continued their "mission goals" without establishing an existence result first, they basically tried to fulfill and democratize the nebulous informal desiderata the earlier forms of healthcare aspired to... It's historically grown holy-grail level provable unobtainium. Its a pointless crusade worse than fighting windmills.
Can’t agree with the premise. It seems that through changing regulatory regimes and technology a 1 in 10 chance is the economic optimum for this. Interesting stability, certainly, but I’d expect that as technology improves success rate, funding increases until the marginal project is unprofitable.
The better we get at doing things, the more ambitious we get. As an example, we can keep babies alive much earlier in gestation, so we try harder if they’re earlier than we would before. We should expect a homeostatic equilibrium between our skill and our ambition.
But technology isn't improving the success rate. And the marginal project is already expected to be unprofitable. Pharma is a hits driven business and the hits are getting simultaneously more expensive to produce, harder to find, and enter into a much more competitive market environment.
The economic calculus is ambivalent between a clinical trial with a 10% chance of a drug discovery worth a billion dollars, a 100% chance of a drug discovery worth a hundred million dollars, and a 1% chance of a drug discovery worth ten billion dollars. I'm not sure the economically marginal project is less likely to fail as much as it would simply target more niche markets.
I 100% agree. the criteria for sucess is something that is missed by simplyifying things to a number.
Those failures can include drugs that would be considerer miraculous 10 or 20 years ago. You might have a drug that functionally cures HIV, but failed to outperform the standard of care in a RCT.
You have drugs like Lorbrena for non-small cell lung cancer where the median life extension is unknown because >50% of patients are alive 7 years out. Or Keytruda where 50% of advanced melenoma patients appear functionally cured 10 years post treatment.
These are examples of ambitious drugs that wouldn’t have been attempted even a few decades earlier. But technology improved and the money was there so they tried anyway, and found success. But for any 3 successes there are still 27 failures because those 27 were equally ambitious.
Feels like a just-so story. How to quantify “ambitious”? If the goal were flight, would jumping off cliffs with wings like birds be “ambitious”?
The most parsimonious explanation is either our models or methods (or both) are garbage. Something’s missing. The failure rate is insane, and writing it off as ambition or “biology is hard” rather than digging in does us no favors.
"Why would anyone make a drug if the current prevalent treatment was as good as a functional cure?"
I am not sure whether you ask for economic why, or an overall why.
The overall why seems easier to explain:
a. The new drug may be cheaper.
b. The new drug may be safer. (e.g. no risk of anaphylaxis etc.)
c. The new drug may not need cold storage (huge problem outside the First World).
d. The new drug may have other properties that the original does not have (e.g. being taken once a day instead of four times a day, not requiring people to stay off specific food etc.)
e. The side effects (such as vomiting) may be lower.
In general, it is always better to have alternatives in medicine.
This is very facile thinking - are there any examples of a-e in the real world (namely, a brand new drug program was spun up to exploit a-e) ? I think not, which is why I asked the question.
IIRC there are serious attempts at "vaccination plasters", which would solve the cold storage problems of classical vaccines. Some really bad infections survive in reservoirs far from the closest hospital with reliable electricity.
And frankly your comment about "facile thinking" is totally unwelcome. Have you seen someone vomiting their guts out after chemo? That is torture.
Oncology is one field where every less punishing treatment would be useful. Some patients have to be taken off otherwise life-saving treatments because they cannot tolerate them. That is true even with well-treatable cancers such as Hodgkins, which 90+ per cent of patients survive.
Keytruda revenue is 30+ billion per year.
If your new drug performs 1% better than Keytruda, that revenue becomes yours.
If your drug perfroms 1% worse, you become a failure in the statistics.
That isn’t how a rational person would pursue this - it costs $1B to make a drug start to finish, and there’s no reason to believe your new mechanism will work.
If you’re talking about copycat mechanisms, that is fine but we’re still left discussing successes when OP is about failures
Many comments so far seem to try to handwave away the 90 % failure as somehow "optimal" in the system, which seems absurd to me. It is clearly not advantageous for individual companies to keep a drug candidate alive long enough for it to fail in stage III or IV. One obvious question is why they don't and it is very, very tempting to speculate that it's because the problems are getting harder, we are targeting novel mechanisms etc. Again, I think this misses a simpler explanation:
As with many cases where companies make seemingly bad decisions, I think a lot of the explanation lies in system dynamics. Think about the incentive structure inside large pharma companies - it is generally not a career advancement move for a project manager to kill the drug candidate they oversee. It is career advancing to get it approved for the next stage. What could possibly go wrong in this world?
> it is generally not a career advancement move for a project manager to kill the drug candidate they oversee.
You're grossly oversimplifying the process. The decision to "kill" a drug is huge, especially if it's already in the clinic (per the article). That decision will be taken by a large group of people, not an individual - and certainly not a "project manager".
Sure, but Pharma A is full of career managers who never kill drugs under development and Upstart B relentlessly culls drugs that don't work early on. Upstart B's failure rate at the final stages is under 50% so they develop 5 times as many drugs, beating the existing company, and indeed all other drug companies.
Since this isn't happening, it seems this might not be the explanation.
For me it seems this simplification falls flat right away when you take into account budget constraints.
It is not software development where you can start a project every month see how it goes and drop it or pivot.
I guess they use a lot of computer aided models before they even start serious parts but I believe this discussion is not about failure rates on that stage because then it would be 99%
The human body is an extremely complex system. Simulating it completely accuracy would require a computer many orders of magnitude more powerful than anything currently existing, so the only way to know if a treatment doesn't produce any unexpected side effect is years of empirical testing, because such things can take years to manifest. Fundamentally the problem space contains inescapable complexity; it's not the fault of pharma firms.
I'm actually surprised it isn't going up over time. That is naively what you would expect as the low-hanging fruit is plucked. So the fact that it's been stable is probably a sign that scientific advances are roughly keeping pace with the (presumably) increasing challenge of finding ever more targets for drugs.
I actually do this for a living within pharma now! There's a TON of work that goes into drug discovery before we even call it a program. The odds of success are low, so we put in months of work evaluating a candidate before even have a hunch of a program.
Yes, the science advances, previously high-hanging fruits become low-hanging become high-hanging again [1], but the tooling also advances: we now have databases like OpenTargets which let us more easily evaluate potential drug targets. Failure is so much more than that, though: a program can fail after you've shown efficacy in animals, sometimes it just doesn't happen in the human subjects. Or you fail to find the right measurement (endpoint). A million ways to die.
[1] Gene editing is an example: impossible, then very possible, but now the blocker is public perception which in turn blocks investment.
I suspect there's a ceiling on failure rates people are willing to operate under. A diversified investor might still be willing to take high risk/reward bets. But workers can go their whole career producing nothing of value. It becomes hard to tell the difference between smart, hard working people who were unlucky and lazy grifters. I've done 2 series-A startups and I don't think I can do another one.
He points out that it basically has gone up, since the 80s and 90s brought some new therapeutic targets and approaches that by now have been mined out.
I had to look what "clinical failure rate" means and I think I got the answer in this paper [1] and... I'm not going to say that it is worrisome. Unlike other industries, we can't accurately model in the pre-prototype phase how something will behave in living beings. And to use the author examples, cars and airliners are big and way simpler to model than the complex pathways that you find in biology.
We are chipping away at that problem but it's not like we have it mostly solved as other engineering areas of knowledge. I would argue that due that phase I successes isn't the benchmark, but phase II success should be the actual measure. I'm sure that if someone charts the accumulative success rate for each phase of clinical trials, you will see that phase I is the most brutal one.
Isn't this the entire point of vc culture? Builders are rubes; investors are the smart kids who can cash out before their investment collapses.
The missing ingredient of course is the people. We cannot let this country pretend it's functional when it invests so poorly. Any sane version of the us will expect years if nit decades of economic pain before these investments pay off
As someone who works in research, this isn’t surprising at all. It’s hard to find something that treats (well most often reduces symptoms) of a particular disease. It’s even more difficult to find something that is also safe at the dose level it takes to treat said disease. Are there faster ways to do this? Probably not. AI is only going to help out so much, just like automated drug screening only helped so much since its introduction in the 90s.
>So when we do get something to work and something that people are willing to pay money for, we try to squeeze every dollar out of it because we never know when the next one will come along.
Would make a small correction. It does not have to be people. It can be also be "doctors" or "governments". It is easier to convince or coerce/fool a lesser number of humans (doctors) or a single government regulatory body than to fool/convince every one who use the product, because the people can directly evaluate the product.
And when a single doctor is coerced, then the product is forced on hundreds of their patients. When a government is coerced, then the product is forced on tens of millions of people..
But let’s think about that 91% failure rate for a moment. When I bring this up in presentations, I invite the audience to consider what the auto industry would look like of 91% of new car designs proved unable to roll out of the factory, or if 91% of new airliner models were unable to leave the ground
Is an utterly irrelevant comparison. For physical thing we have engineering, and the practical application of the trades and craft, trial and error.
For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
Indeed, I suspect the failure rate of, say, new jet engine designs is rather high as well -- those failures just never get reported in a federal repository, unlike RCTs, since they never make it out of the simulator or the prototyping lab. And we have, comparatively, much better computational models of how airplanes fly than how cancer cells mutate. FWIW this clinical stat is far better than Edison's supposed lightbulb-idea failure rate!
Jet engine design is iterative, and the basic principles are well-understood.
Drug design seems a lot more binary. You can find a new pathway, but drugs themselves are fairly simply molecules, and you can't iteratively 'fix bugs' the way you can in an engine or a piece of software.
The engine is a given, it's almost astronomically complicated, you know a lot less about how it works than you'd like to, and you're trying to change how it works while it's running without breaking anything, using tiny rigid parts that have to snap into place correctly and can't be bent to fit.
This is a pretty outdated view, particularly with respect to biologics, which are some of the most complex structures humans fabricate. Many are in fact quite similar to iterative prototypes of jet engines.
You might have something like Anti–vascular endothelial growth factor therapy, where you have a binding target structure, and iterate the uses and molecular structure around it, or combine it with other structures and binding sites.
You might go from mab to fab, or to bispecific mab using CrossMab IgG architecture, or bispecific fab using dutafab fragment architechture. This analogizes to mixing and matching different jet engine technologies into different platofrms.
VEGF targeting biologics have netted >150 billion dollars to date, and this will only grow faster in the future.
iterative GLP-1 technologies will be much the same, where people are literally iteratively fixing bugs.
The development pipeline for both is littred with failed iterations.
Also, highly related, in the previous paragraph, they say: "We know a lot more about the biology of disease - although God knows, not nearly enough".
Anyone smarter than a 10 year old would not we actually know almost everything about car design. How it works is not at all hidden. We have iterated on pretty much the same thing for 100 years. The current state is about efficiency, materials and manufacturing.
When you have deep knowledge and experience in a field you will get near 100%.
Targeted gene therapy, cancer survival rates, trauma care, the advancements in hip and knee replacement, all sorts of surgery, HIV is now a non-issue with the right care. The list goes on.
Drug development is a hard problem because the solution space is poorly constrained: biochemistry is complex and messy, expecting one chemical substance to have narrow positive effects is probably hopeless.
So the claim is that we're out of the wild west because there's a set of things we've made advancements on, despite drug development getting harder and harder?
> Is an utterly irrelevant comparison. For physical thing we have engineering, and the practical application of the trades and craft, trial and error.
>
> For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
I don't think its irrelevant, but a better comparison would be to what vacuum tube development looked like before we understood electrons. There were some very whacky designs and most of them didn't work for crap.
And, yeah, I would argue that PCR shifted us from the alchemy phase of biology to the science phase and now mRNA has shifted us from the science phase of biology to t he engineering phase of biology. We're just getting started.
Without doing any serious thought, it is concerning that the statistic doesn't vary more temporally. It seems too consistent, and raises the suspicion that it reflects regulatory agenda more than anything intrinsic
You want high failure rates. If failure rates were low, that tells you that you are probably being far too conservative in funding new clinical trials.
> I’m fond of saying that the most important single statistic about the drug industry is the clinical failure rate, which is (by any reasonable standard) appallingly high.
He starts off on the wrong note, there is nothing at all wrong with a high rate of clinical failures. If anything, the reasonable argument standard might be that this rate is too low. It implies researchers are trying things that they expect to have a 10% chance of working out. That means we're missing out on all the cures and techniques that have a 1% chance of working out but but nonetheless do work.
Failed attempts cost society nearly nothing and successes will have compounding benifits for, y'know, lets optimistically say the human race survives for centuries. 1% or 0.1% success rates sound completely reasonable with that sort of lopsided risk profile. There isn't much of a reason not to try anything and everything that has the faintest chance of helping and see what happens.
Rather notably, companies are profit-driven, not good-for-society driven. If 99% of your attempts which cost millions to do fail, the company either has to raise the prices of the drug insanely high or not work as a company. In addition, most drugs aren't humanity saving or crucial for humans to live. You could solve all forms of cancer tomorrow, which would be a great thing, but you aren't "saving humanity". You add less than a decade or so, maximum, to the average human lifespan. This isn't critical to humanity's existence, so this "try or die" mentality doesn't work.
If this try or die mentality does work, why don't we try every possible amino acid chain in every configuration possible? It's an infinitesimally small chance but try or die "outweighs". This is evidently absurd.
This goes to show why private r and d is not going to get us that star trek future. Private capital is too risk adverse and short term in outlook for a lot of potential innovation. Interesting to think about how stifled our innovation potential is these days, given that things are expected to be immediately and massively profitable if they are to be invested into.
Lowe mentions costs 0 times though, he isn't making an economic argument. If you want to make an economic argument then 90% is probably the consensus on what is optimal given that is what people are doing.
> If this try or die mentality does work, why don't we try every possible amino acid chain in every configuration possible?
You tell me. If someone wants to start working through the amino acids one by one I'm not going to say they should be stopped; sincerest good luck to them. I'm just not going to be the one funding it.
> You could solve all forms of cancer tomorrow, which would be a great thing, but you aren't "saving humanity".
This is a straw man. "Saving humanity", whatever that means, is outside the scope of medicine and I never argued for it. Nobody did. It isn't relevant.
For a great example of an extreme success rate being a bad thing, look at Japan's legal system. Prosecutors only take cases that they think they will win, and that assumption is now baked into the system, to the point that they are no longer very good at prosecuting, and anyone that is prosecuted is effectively presumed guilty.
Except that governing bodies approve many of these drugs which, further down the line, consumers may experience high failure rates or even toxicity, which in other industries such results would be considered outright fraud subject to legal action. if sold to consumers.
As someone who used to work in pharmaceutical R&D, the important thing to remember is the hurdle to get over hasn’t remained constant.
The FDA has gotten significantly more strict in its review than it was in the 1960s.
A good example is hERG inhibition as an off-target effect. It’s a receptor on heart muscles and will result in QT prolongation and potential arrhythmias.
It wasn’t discovered until the 1990s. Now every molecule is screened and many are dumped. The impact can vary but there are tons of drugs on the market now that are hERG inhibitors (many discovered after the fact).
It’s a good example of the increased rigor that the FDA applies to everything they review that past trials never had to face.
It’s a similar problem you see with all safety related decisions - as the large risks are eliminated you become more concerned about the smaller risks.
Plenty of successful drugs from the 50’s had hERG activity, yet the impact on safety was marginal. Today plenty of programs are killed over it.
Increase safety scrutiny does increase safety, but at a cost.
There's an alternative that is slowly emerging. Many clinical trials "fail" but the drug candidate in question works really well for identifiable subsets of the participants. Right now pharma companies won't bother pursuing those drugs because they can't market it broadly. But it's still possible these drugs could help people in the future.
Not really an alternative if you can't predict who those candidates are prior to spending a hundred million dollars putting the drug into a broader group of people and then retroactively saying "oh wait now we can 'identify' who it works in."
If they could do that today, they would. Trial criteria are already incredibly narrow specifically to try to encode as much of this knowledge as the company has prior to starting the trial. But it empirically turns out they don't have nearly enough to matter.
It's really not a problem that pre-filtering to that responders group produces too small a market. There are plenty of drug programs going after way-too-small markets because of all these wacky perverse incentives created by insurers and regulators to incentivize the creation of billion-dollar++ drugs for diseases virtually no one has.
I’ve been hearing for 10 years now on how “AI” will change this clinical rate. It’s now like the “This is the year of Linux” prediction meme. And like Derek has mentioned multiple times you can’t statiscialise biology. I don’t think many people get it so we have tech bros like us throw AI at data.
I've been hearing some form of it since I was an undergrad. I just submitted by packet for full professor. There have been some spectacular strides, but we've been through multiple rounds of "Stand aside, the computers will take it from here."
People saying this stuff aren't in the game. ML has been used in clinical research for basically as long as various ML methods have been around. Basically a century now in other words. AI leaders are trying to get money from people who don't understand the methodology such as investors or trump administration officials. Imagine trying to convince people who actually work day in day out with classic ml and deep learning to use your llm goobly goop bot.
If you can think of a way that you can have a much higher chance than 9% of success in getting a useful treatment to market, you can make a boatload of money. Go ahead, reform it.
Not knowing of a way is inequivalent to inexistence of any way. This is basic and apparently some bloggers are bad writers and or political conservatives of the type "there is no alternative".
The alternative is "have a complete mechanistic understanding of biology." People are definitely working on it, but we are nowhere close to as complete an understanding as we need. The evidence for precisely this claim is in the failure rate of clinical trials, into which we encode our best-known understanding of biology and we still end up being wrong 90% of the time. And that's not including all the pre-clinical candidates, which drive our wrongness rate (i.e. knowledge gap) to 99.99999%+
How? He’s saying that he can’t think of any good ones, and in my experience that’s true of most informed people. This is one of those hard problems that needs solutions, and so far the present system is the best anyone has managed.
I don't care what he's "saying that" which is just your biased interpretation. His essay does not warrant the final concluding sentence which would require a different essay thesis. It's just basic high school reading and writing skills.
Again, anyone who cracks a decent success rate can basically print money. Unfortunately, despite lots of efforts in this area, we have not been able to get the clinical success rate up.
It's pretty low effort to jeer from the outside and say "do better" for an industry that's been trying so many approaches to do so, without any ideas yourself.
The techniques have gotten a lot more sophisticated, but unfortunately a lot of the low hanging, "easy" fruit have been used up. Biology is hard.
It's not for lack of trying. People are definitely trying to find better ways of finding stronger candidates earlier. People are trying to gain a more precise understanding of disease mechanisms.
There are breakthroughs that I could have never imagined such as mRNA vaccines.
I would have to reread the piece carefully but I don't think "lack of trying by Biopharma" is consistent with the author's main argument. Because he is still being somewhat critical of the way it is currently done (why he cites the other industries). I simply disagree with his final sentence since it requires a different set of essay arguments, and also is subjective in the way say a political centrist might say "there is no alternative to fossil fuels" etc.
Most other industries barely innovate. Most industries are just buying off the shelf components and putting them together into larger widgets. Cars, computers, airplanes, that's the business. The components might get more efficient or performant but they generally do the thing they always did. If you squint it is basically the same car/plane/laptop as x years ago. They worked out the optimal car/plane/laptop shape a while ago and now everything sold is exactly that shape and will be for probably the rest of both of our lives and then some, in those industries.
Life science on the other hand is actually on a bleeding edge. Something actually revolutionary might emerge tomorrow after a grad student has an acid trip tonight.
> But let’s think about that 91% failure rate for a moment. When I bring this up in presentations, I invite the audience to consider what the auto industry would look like of 91% of new car designs proved unable to roll out of the factory, or if 91% of new airliner models were unable to leave the ground - and if you only found that out after spending all the R&D money to build them at full size and trying to fly them. No cutting-edge restaurant could survive if 91% of its innovative dishes proved inedible or outright poisonous. What other industries operate under these bizarre conditions?
This is bizarre, bordering on stupid. Frist of all, there likely is ~90% failure rate of prototypes; I feel that roughly matches my experience in engineering. Of course the design that makes it through the process, testing, refinement, and into mass production is not going to have a 90% failure rate, but that's a _finished product_, whereas clinical tests are just that -- tests. Finished cars are more analogous to individual pills coming out of the factory. And I'm not even sure the analogy would be very meaningful anyways, because we have different requirements for things of different impact and importance. A 10% manufacturing defect rate is fine in forks, but not for fire extinguishers.
It's not that much better in plant breeding: You'll find hundreds upon hundreds of supposedly better varieties tried, and 5 years later, you are lucky to get 1 good one that one would consider commercializing. And that's after years of testing changes in smaller settings. We just don't have models that are predictive enough without planting, and waiting for plants to grow takes time.
A large part of that failure rate is in phase 3. Your comparison with prototypes would be more like drug development before even phase 1. Phase 3 is enormously expensive, much more than the earlier parts. And what's even worse, you don't get to learn all that much from failures in drug development in many cases. You can't just fix the problem and try again, you essentially have to try a completely new molecule.
That's fair, phase 3 testing isn't an early prototype, it's a prototype with many years of effort behind it. But I still think it is _more_ like a prototype than it is like a product coming off an assembly line. Perhaps phase 3 is like the latest prototype from boom (the company trying to bring back super-sonic jets). Those prototypes represent 100s of engineering-years put together, and are not much like the prototypes you'd encounter in consumer product development. I still think 90% failure rate is believable for that. How many significant design iterations has Boom gone through so far? I don't really know, but I could believe it's been 10 significant iterations.
> A large part of that failure rate is in phase 3.
I must ask for some data to support this statement, and also your definition of "a large part".
Yeah why is 10% success bad? What is the tradeoff between spent effort and missed cures if we try to tweak the success rate by killing prototypes earlier in the pipeline? While reading I was expecting a reasoned argument... that was stubbornly not coming around.
It's mostly that we have too many false findings pre-clinical trials, most drug targets validated in models that don't actually hold up in humans, so a huge share of the 90% failure is money and years spent testing candidates that were never going to work. That inflates the number of potential candidates that are likely wrong, due how we select them.
There are ways used right now that are working towards reducing those false findings by looking at actual humans, their biomarkers and whenever or not there's an associated molecule to the condition that we would like to pass onto others. It will not kill prototypes, instead it will discourage us from going through a prototype at all by going for better candidates instead.
The prototypes are generally cheap though: you're not standing up a full production line for them before you find out that they're duds.
The problem is that you model the filtering step as a monolithic step.
Things start with academic literature, and models (qualitative or even numerically quantitative, human comprehensible, or computationally predicted effects, ...). Then academic level testing occurs, resulting in putative results, obtained on animal models, cell / tissue cultures, ... before proceeding to human trials. The failure rates are for this final step. Imagine being in control of some pharma fund, there is a huge stream of putative drugs emerging in the literature, and only limited budget AND limited test-patient slots. On average people in this position succeed in selecting one that performs as predicted only ~10% of the time. Thats not a simple exploration-exploitation trade-off. With so many candidates, assuming proper pre-human experiments, one would expect much better results, and you'd from a financial perspective redirect focus towards those drugs with high confidence from prior forms of non-human testing. Yet we see failure rates 80-90%! From a purely financial perspective, there is a huge incentive to place more selection emphasis on confidence, but either its not happening or institutions (public or private) are systematically dropping the ball.
To make the engineering analogy with design methodology: before testing a new implementation, we have the luxury to preselect implementations depending on their unit tests, considerably improving the success rate for the higher level implementation. Yet for the analogy in drug selection we fail miserably.
I don't believe the failure is proper to the selection process. Its that the true fitness function is unavailable, if we had it would simply be a matter of performing gradient descent.
But obviously earthly biology does not come with a reference manual of all niches, and analytic objective differentiable fitness functions.
To a large extent it is in fact still the exploration-exploitation curve, but a meta level.
We can't bypass natural selection, we can't turbocharge natural selection (like the Nazi's tried), not only because it is evil, but because the fitness statistic is for all purposes and intents, emergent in nature.
The average reader here will be very familiar with the concept of premature optimization: don't start micro optimizing your code in assembler before functional correctness, first go for correctness, then reason about the hot paths from profiling and investigate and optimize from there.
Historically, hospitals and medicine long predate modern science and biology.
It predates the discovery of natural selection, it predates the measurement of selection phenomena on shortlived organisms.
Healthcare is high inertia never-to-seldom-recognize-earlier-mistakes domain.
The justification of healthcare is never supported by some axiomatized formally verifiable system of ethics, it is justified on the basis of associations and vague nebulous historically grown rules.
Socialized healthcare is a world-widely supported doctrine. Don't wish unto others what you wouldn't wish onto yourself is another. There is some nebulous concept of a right to healthcare, but a right to what: some nebulous default "healthy" state? There is also a strong connection with egalitarianism, if someone gets sick from a flu we somehow believe it is desirable to help them overcome it, because we claim this flu could have afflicted any individual equally.
THe healthcare zooko's triangle looks like this, you can't have all 3 of the following, so once you have unconditionally selected one desideratum, you will be left torn by the decision between only one of the remaining desiderata:
1. egalitarian access to healthcare 2. working healthcare solutions towards "individual health", i.e. improving procreation rates statistically 3. maintained fitness of the collective human genome distribution
Assuming without compromise 1: egalitarian access to healthcare: ===========
To the extent a healthcare instrument (drugs, or tools like glasses) works (2), it undoes decreased genetic procreation rates, inducing a higher incidence rate in future generations, reducing the fitness of the human genome (3): we have not cured or treated this individual, we have traded innate health, fitness and quality of life of future generations for the convenience and comfort of individuals in the current generations.
To the extend we insist to maintain genetic fitness, it would require the healthcare instrument to be ineffective and thus not influence procreation statistics of fertile individuals. To the extent a healthcare instrument doesn't affect the procreation statistics of a fertile individual, the healthcare instrument didn't improve quality of life for this individual (and is effectively a quack measure). For example without the flu shot, an fertile individual might have met a potential mate, or been in the mood to mate with an already associated partner, but an individual without the flu shot might have felt too sick, or been to repulsive for a mate or partner during sickness. Natural selection first and foremost is about procreation probabilities and rates, and much rarer the individually stronger but collectively weaker signal of death.
For example: there is wide consensus that pre-modern tribal hunter/gatherer humans only had sub 5% incidence rates of poor vision requiring corrective glasses according to modern standards. The selective pressure on eyesight was so strong that even 1 or 2 generations of modern medicine, results in large majority of population requiring prescription glasses, the loss of a strong selective pressure quickly results in drift away from fitness.
Assuming without compromise 2. "individually effective" healthcare instruments: =============
If we select to keep (1) egalitarian access then it will be at the expense of (3) maintained fitness in the humanities collective genome: the egalitarian access to the individually effective healthcare measure, will result in the loss of genetic utility, since the utility is supported and provided externally instead of innately.
If we select to maintain the collective genetic fitness of humanity (3), it can be attained while maintaining "individually effective healthcare" measures (drugs, crutches, pacemakers,...), but only if we relax (1) egalitarian access to healthcare: a modern nation state can admit migration or otherwise support the transfer of genetic material from regions with no or little healthcare, or from regions where such healthcare was only recently introduced. Steady state reliance on such a stream of wild-type humans, is basically bio-colonialism: it requires a region where humans face natural selection without help from healthcare, and their genes are used to improve or maintain fitness of modern nation states elsewhere.
Assuming without compromise (3) maintaining fitness of humanities collective genome:
If we insist on maintaining (1) egalitarian access; then it will be at the cost of (2) healthcare that objectively improves the quality of life and thus procreation statistics in the case of fertile individuals. So we could all enjoy egalitarian access to non-functional medicine.
If we insist on maintaining (2) individually effective healthcare measures, which improve quality of life and hence procreation statistics of fertile individuals, we have to sacrifice (1) egalitarian access: if a sufficient collection of humans is basically deprived of healthcare access, then yes we could maintain fitness by selecting their genetics for procreation.
It's a veritable zooko's triangle, and an absolute nightmare once comprehension sinks in.
If for every QAPR (quality adjusted procreation rate) were taken into account for healthcare interventions, they'd all go negative! whatever current "meritocratic assessment" of healthcare instruments is basically fraud from the perspective of genetics and natural selection, like those video's you see from parents climbing school walls in India to help their children cheat on some national level exams, and somehow this being normalized by society...
It is not a question of allegiance with or against modern healthcare, it is a question of internal consistency of what is known about natural selection, selection pressure, healthcare, etc. We only know how to trade with loss (but you don't need a degree in medicine to attain that skill).
Modern medicine stems from a premature optimization objective (and the original goals did not involve (1) egalitarian access -for nobility and their armies- nor did the original goals involve (3) maintaining genetic fitness for future generations -the statistics of natural selection and concepts like selection pressure were only poorly understood).
Socialized healthcare institutions continued their "mission goals" without establishing an existence result first, they basically tried to fulfill and democratize the nebulous informal desiderata the earlier forms of healthcare aspired to... It's historically grown holy-grail level provable unobtainium. Its a pointless crusade worse than fighting windmills.
Can’t agree with the premise. It seems that through changing regulatory regimes and technology a 1 in 10 chance is the economic optimum for this. Interesting stability, certainly, but I’d expect that as technology improves success rate, funding increases until the marginal project is unprofitable.
The better we get at doing things, the more ambitious we get. As an example, we can keep babies alive much earlier in gestation, so we try harder if they’re earlier than we would before. We should expect a homeostatic equilibrium between our skill and our ambition.
But technology isn't improving the success rate. And the marginal project is already expected to be unprofitable. Pharma is a hits driven business and the hits are getting simultaneously more expensive to produce, harder to find, and enter into a much more competitive market environment.
The economic calculus is ambivalent between a clinical trial with a 10% chance of a drug discovery worth a billion dollars, a 100% chance of a drug discovery worth a hundred million dollars, and a 1% chance of a drug discovery worth ten billion dollars. I'm not sure the economically marginal project is less likely to fail as much as it would simply target more niche markets.
Care to calculate the marginal utility of torching your credibility?
I 100% agree. the criteria for sucess is something that is missed by simplyifying things to a number.
Those failures can include drugs that would be considerer miraculous 10 or 20 years ago. You might have a drug that functionally cures HIV, but failed to outperform the standard of care in a RCT.
You have drugs like Lorbrena for non-small cell lung cancer where the median life extension is unknown because >50% of patients are alive 7 years out. Or Keytruda where 50% of advanced melenoma patients appear functionally cured 10 years post treatment.
This comment makes no sense.
Why would anyone make a drug if the current prevalent treatment was as good as a functional cure?
What do lorbrena and keytruda have to do with the rate of failures being constant?
Lots of reasons.
The new drug could be cheaper to manufacture, fewer side effects, a full cure in stead of a functional cure.
Cost of manufacture is not a meaningful driver of pricing outside the biological
How will your clinical trial of the ostensible full cure work if standard of care is curing people?
These are examples of ambitious drugs that wouldn’t have been attempted even a few decades earlier. But technology improved and the money was there so they tried anyway, and found success. But for any 3 successes there are still 27 failures because those 27 were equally ambitious.
Feels like a just-so story. How to quantify “ambitious”? If the goal were flight, would jumping off cliffs with wings like birds be “ambitious”?
The most parsimonious explanation is either our models or methods (or both) are garbage. Something’s missing. The failure rate is insane, and writing it off as ambition or “biology is hard” rather than digging in does us no favors.
"Why would anyone make a drug if the current prevalent treatment was as good as a functional cure?"
I am not sure whether you ask for economic why, or an overall why.
The overall why seems easier to explain:
a. The new drug may be cheaper.
b. The new drug may be safer. (e.g. no risk of anaphylaxis etc.)
c. The new drug may not need cold storage (huge problem outside the First World).
d. The new drug may have other properties that the original does not have (e.g. being taken once a day instead of four times a day, not requiring people to stay off specific food etc.)
e. The side effects (such as vomiting) may be lower.
In general, it is always better to have alternatives in medicine.
This is very facile thinking - are there any examples of a-e in the real world (namely, a brand new drug program was spun up to exploit a-e) ? I think not, which is why I asked the question.
IIRC there are serious attempts at "vaccination plasters", which would solve the cold storage problems of classical vaccines. Some really bad infections survive in reservoirs far from the closest hospital with reliable electricity.
And frankly your comment about "facile thinking" is totally unwelcome. Have you seen someone vomiting their guts out after chemo? That is torture.
Oncology is one field where every less punishing treatment would be useful. Some patients have to be taken off otherwise life-saving treatments because they cannot tolerate them. That is true even with well-treatable cancers such as Hodgkins, which 90+ per cent of patients survive.
Keytruda revenue is 30+ billion per year. If your new drug performs 1% better than Keytruda, that revenue becomes yours. If your drug perfroms 1% worse, you become a failure in the statistics.
That isn’t how a rational person would pursue this - it costs $1B to make a drug start to finish, and there’s no reason to believe your new mechanism will work.
If you’re talking about copycat mechanisms, that is fine but we’re still left discussing successes when OP is about failures
Why isnt it rational?
You spend the 1 billion for a ticket to win 30 billion/yr for the next several years.
Is it rational to spend 1$ to make $100 10% of the time? A 99% failure rate would be break even.
Many comments so far seem to try to handwave away the 90 % failure as somehow "optimal" in the system, which seems absurd to me. It is clearly not advantageous for individual companies to keep a drug candidate alive long enough for it to fail in stage III or IV. One obvious question is why they don't and it is very, very tempting to speculate that it's because the problems are getting harder, we are targeting novel mechanisms etc. Again, I think this misses a simpler explanation:
As with many cases where companies make seemingly bad decisions, I think a lot of the explanation lies in system dynamics. Think about the incentive structure inside large pharma companies - it is generally not a career advancement move for a project manager to kill the drug candidate they oversee. It is career advancing to get it approved for the next stage. What could possibly go wrong in this world?
> it is generally not a career advancement move for a project manager to kill the drug candidate they oversee.
You're grossly oversimplifying the process. The decision to "kill" a drug is huge, especially if it's already in the clinic (per the article). That decision will be taken by a large group of people, not an individual - and certainly not a "project manager".
Sure, but Pharma A is full of career managers who never kill drugs under development and Upstart B relentlessly culls drugs that don't work early on. Upstart B's failure rate at the final stages is under 50% so they develop 5 times as many drugs, beating the existing company, and indeed all other drug companies.
Since this isn't happening, it seems this might not be the explanation.
For me it seems this simplification falls flat right away when you take into account budget constraints.
It is not software development where you can start a project every month see how it goes and drop it or pivot.
I guess they use a lot of computer aided models before they even start serious parts but I believe this discussion is not about failure rates on that stage because then it would be 99%
The human body is an extremely complex system. Simulating it completely accuracy would require a computer many orders of magnitude more powerful than anything currently existing, so the only way to know if a treatment doesn't produce any unexpected side effect is years of empirical testing, because such things can take years to manifest. Fundamentally the problem space contains inescapable complexity; it's not the fault of pharma firms.
I'm actually surprised it isn't going up over time. That is naively what you would expect as the low-hanging fruit is plucked. So the fact that it's been stable is probably a sign that scientific advances are roughly keeping pace with the (presumably) increasing challenge of finding ever more targets for drugs.
I actually do this for a living within pharma now! There's a TON of work that goes into drug discovery before we even call it a program. The odds of success are low, so we put in months of work evaluating a candidate before even have a hunch of a program.
Yes, the science advances, previously high-hanging fruits become low-hanging become high-hanging again [1], but the tooling also advances: we now have databases like OpenTargets which let us more easily evaluate potential drug targets. Failure is so much more than that, though: a program can fail after you've shown efficacy in animals, sometimes it just doesn't happen in the human subjects. Or you fail to find the right measurement (endpoint). A million ways to die.
[1] Gene editing is an example: impossible, then very possible, but now the blocker is public perception which in turn blocks investment.
I suspect there's a ceiling on failure rates people are willing to operate under. A diversified investor might still be willing to take high risk/reward bets. But workers can go their whole career producing nothing of value. It becomes hard to tell the difference between smart, hard working people who were unlucky and lazy grifters. I've done 2 series-A startups and I don't think I can do another one.
He points out that it basically has gone up, since the 80s and 90s brought some new therapeutic targets and approaches that by now have been mined out.
I had to look what "clinical failure rate" means and I think I got the answer in this paper [1] and... I'm not going to say that it is worrisome. Unlike other industries, we can't accurately model in the pre-prototype phase how something will behave in living beings. And to use the author examples, cars and airliners are big and way simpler to model than the complex pathways that you find in biology.
We are chipping away at that problem but it's not like we have it mostly solved as other engineering areas of knowledge. I would argue that due that phase I successes isn't the benchmark, but phase II success should be the actual measure. I'm sure that if someone charts the accumulative success rate for each phase of clinical trials, you will see that phase I is the most brutal one.
1: https://www.nature.com/articles/nrd.2016.136
> I had to look what "clinical failure rate" means
That should have been the end if the comment; the follow-up by demonstrating such expertise just has me laughing.
Roughly equivalent to the percentage of startups that fail.
Lower numbers don’t mean we’re doing better, it means we’re trying less.
I’m not entirely sure other startups necessarily want to succeed…
Their goal often seems not to be toward a successful, sustainable product. They just market their vision in the hopes of being acquired.
A few fewer failures might not be entirely a bad thing.
Isn't this the entire point of vc culture? Builders are rubes; investors are the smart kids who can cash out before their investment collapses.
The missing ingredient of course is the people. We cannot let this country pretend it's functional when it invests so poorly. Any sane version of the us will expect years if nit decades of economic pain before these investments pay off
lower numbers might be play to not lose, instead of play to win. On the other hand, do no harm.
ugh, I don't know
As someone who works in research, this isn’t surprising at all. It’s hard to find something that treats (well most often reduces symptoms) of a particular disease. It’s even more difficult to find something that is also safe at the dose level it takes to treat said disease. Are there faster ways to do this? Probably not. AI is only going to help out so much, just like automated drug screening only helped so much since its introduction in the 90s.
The author, Derek Lowe, also writes the hilarious Things I won't work with series (https://www.science.org/content/blog-post/things-i-won-t-wor...)
>So when we do get something to work and something that people are willing to pay money for, we try to squeeze every dollar out of it because we never know when the next one will come along.
Would make a small correction. It does not have to be people. It can be also be "doctors" or "governments". It is easier to convince or coerce/fool a lesser number of humans (doctors) or a single government regulatory body than to fool/convince every one who use the product, because the people can directly evaluate the product.
And when a single doctor is coerced, then the product is forced on hundreds of their patients. When a government is coerced, then the product is forced on tens of millions of people..
Doctors are people
But they are not the one paying for the drugs they prescribe...
This is completely unsurprising, and this:
But let’s think about that 91% failure rate for a moment. When I bring this up in presentations, I invite the audience to consider what the auto industry would look like of 91% of new car designs proved unable to roll out of the factory, or if 91% of new airliner models were unable to leave the ground
Is an utterly irrelevant comparison. For physical thing we have engineering, and the practical application of the trades and craft, trial and error.
For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
Indeed, I suspect the failure rate of, say, new jet engine designs is rather high as well -- those failures just never get reported in a federal repository, unlike RCTs, since they never make it out of the simulator or the prototyping lab. And we have, comparatively, much better computational models of how airplanes fly than how cancer cells mutate. FWIW this clinical stat is far better than Edison's supposed lightbulb-idea failure rate!
Jet engine design is iterative, and the basic principles are well-understood.
Drug design seems a lot more binary. You can find a new pathway, but drugs themselves are fairly simply molecules, and you can't iteratively 'fix bugs' the way you can in an engine or a piece of software.
The engine is a given, it's almost astronomically complicated, you know a lot less about how it works than you'd like to, and you're trying to change how it works while it's running without breaking anything, using tiny rigid parts that have to snap into place correctly and can't be bent to fit.
High failure rates aren't surprising.
This is a pretty outdated view, particularly with respect to biologics, which are some of the most complex structures humans fabricate. Many are in fact quite similar to iterative prototypes of jet engines.
You might have something like Anti–vascular endothelial growth factor therapy, where you have a binding target structure, and iterate the uses and molecular structure around it, or combine it with other structures and binding sites.
You might go from mab to fab, or to bispecific mab using CrossMab IgG architecture, or bispecific fab using dutafab fragment architechture. This analogizes to mixing and matching different jet engine technologies into different platofrms.
VEGF targeting biologics have netted >150 billion dollars to date, and this will only grow faster in the future.
iterative GLP-1 technologies will be much the same, where people are literally iteratively fixing bugs.
The development pipeline for both is littred with failed iterations.
> For physical thing we have engineering, and the practical application of the trades and craft, trial and error.
> For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
You could say the same about deep learning. Yet we see improvements every day.
Also, highly related, in the previous paragraph, they say: "We know a lot more about the biology of disease - although God knows, not nearly enough".
Anyone smarter than a 10 year old would not we actually know almost everything about car design. How it works is not at all hidden. We have iterated on pretty much the same thing for 100 years. The current state is about efficiency, materials and manufacturing.
When you have deep knowledge and experience in a field you will get near 100%.
> For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
Why do you say that? What's the evidence? We continue to have virtually no clue how to make drugs, per TFA.
Targeted gene therapy, cancer survival rates, trauma care, the advancements in hip and knee replacement, all sorts of surgery, HIV is now a non-issue with the right care. The list goes on.
Drug development is a hard problem because the solution space is poorly constrained: biochemistry is complex and messy, expecting one chemical substance to have narrow positive effects is probably hopeless.
...?
So the claim is that we're out of the wild west because there's a set of things we've made advancements on, despite drug development getting harder and harder?
If someone said, 91% of software projects fail, i would beieve them.
> Is an utterly irrelevant comparison. For physical thing we have engineering, and the practical application of the trades and craft, trial and error. > > For modern medicine, we're only just starting to come out of the wild west era. Or perhaps slightly further along than that.
I don't think its irrelevant, but a better comparison would be to what vacuum tube development looked like before we understood electrons. There were some very whacky designs and most of them didn't work for crap.
And, yeah, I would argue that PCR shifted us from the alchemy phase of biology to the science phase and now mRNA has shifted us from the science phase of biology to t he engineering phase of biology. We're just getting started.
Without doing any serious thought, it is concerning that the statistic doesn't vary more temporally. It seems too consistent, and raises the suspicion that it reflects regulatory agenda more than anything intrinsic
Going from an 80% to a 90% failure rate means the success rate halved; that's a pretty big change in my book.
You want high failure rates. If failure rates were low, that tells you that you are probably being far too conservative in funding new clinical trials.
> I’m fond of saying that the most important single statistic about the drug industry is the clinical failure rate, which is (by any reasonable standard) appallingly high.
He starts off on the wrong note, there is nothing at all wrong with a high rate of clinical failures. If anything, the reasonable argument standard might be that this rate is too low. It implies researchers are trying things that they expect to have a 10% chance of working out. That means we're missing out on all the cures and techniques that have a 1% chance of working out but but nonetheless do work.
Failed attempts cost society nearly nothing and successes will have compounding benifits for, y'know, lets optimistically say the human race survives for centuries. 1% or 0.1% success rates sound completely reasonable with that sort of lopsided risk profile. There isn't much of a reason not to try anything and everything that has the faintest chance of helping and see what happens.
Rather notably, companies are profit-driven, not good-for-society driven. If 99% of your attempts which cost millions to do fail, the company either has to raise the prices of the drug insanely high or not work as a company. In addition, most drugs aren't humanity saving or crucial for humans to live. You could solve all forms of cancer tomorrow, which would be a great thing, but you aren't "saving humanity". You add less than a decade or so, maximum, to the average human lifespan. This isn't critical to humanity's existence, so this "try or die" mentality doesn't work.
If this try or die mentality does work, why don't we try every possible amino acid chain in every configuration possible? It's an infinitesimally small chance but try or die "outweighs". This is evidently absurd.
This goes to show why private r and d is not going to get us that star trek future. Private capital is too risk adverse and short term in outlook for a lot of potential innovation. Interesting to think about how stifled our innovation potential is these days, given that things are expected to be immediately and massively profitable if they are to be invested into.
Lowe mentions costs 0 times though, he isn't making an economic argument. If you want to make an economic argument then 90% is probably the consensus on what is optimal given that is what people are doing.
> If this try or die mentality does work, why don't we try every possible amino acid chain in every configuration possible?
You tell me. If someone wants to start working through the amino acids one by one I'm not going to say they should be stopped; sincerest good luck to them. I'm just not going to be the one funding it.
> You could solve all forms of cancer tomorrow, which would be a great thing, but you aren't "saving humanity".
This is a straw man. "Saving humanity", whatever that means, is outside the scope of medicine and I never argued for it. Nobody did. It isn't relevant.
Curing all cancer would be bad for the world, because 90% of those cancer kills are old people.
For a great example of an extreme success rate being a bad thing, look at Japan's legal system. Prosecutors only take cases that they think they will win, and that assumption is now baked into the system, to the point that they are no longer very good at prosecuting, and anyone that is prosecuted is effectively presumed guilty.
Except that governing bodies approve many of these drugs which, further down the line, consumers may experience high failure rates or even toxicity, which in other industries such results would be considered outright fraud subject to legal action. if sold to consumers.
> Failed attempts cost society nearly nothing
The world does not have unlimited resources. Money spent on one project is money that can’t be spent on another.
Failures also have lots of valuable information. There was a recent Complex Systems podcast about efforts to glean pharma trials.
https://www.complexsystemspodcast.com/episodes/ruxandra-tesl...
If failure rates went down wouldn't that increase the number of clinical trials until the failure rates go back up.
Doing hard things is hard.
As someone who used to work in pharmaceutical R&D, the important thing to remember is the hurdle to get over hasn’t remained constant.
The FDA has gotten significantly more strict in its review than it was in the 1960s.
A good example is hERG inhibition as an off-target effect. It’s a receptor on heart muscles and will result in QT prolongation and potential arrhythmias.
It wasn’t discovered until the 1990s. Now every molecule is screened and many are dumped. The impact can vary but there are tons of drugs on the market now that are hERG inhibitors (many discovered after the fact).
It’s a good example of the increased rigor that the FDA applies to everything they review that past trials never had to face.
It's interesting that you frame it as the FDA increasing regulation and not that science has increased its foreknowledge of problems.
It’s a similar problem you see with all safety related decisions - as the large risks are eliminated you become more concerned about the smaller risks.
Plenty of successful drugs from the 50’s had hERG activity, yet the impact on safety was marginal. Today plenty of programs are killed over it.
Increase safety scrutiny does increase safety, but at a cost.
There's an alternative that is slowly emerging. Many clinical trials "fail" but the drug candidate in question works really well for identifiable subsets of the participants. Right now pharma companies won't bother pursuing those drugs because they can't market it broadly. But it's still possible these drugs could help people in the future.
Not really an alternative if you can't predict who those candidates are prior to spending a hundred million dollars putting the drug into a broader group of people and then retroactively saying "oh wait now we can 'identify' who it works in."
If they could do that today, they would. Trial criteria are already incredibly narrow specifically to try to encode as much of this knowledge as the company has prior to starting the trial. But it empirically turns out they don't have nearly enough to matter.
It's really not a problem that pre-filtering to that responders group produces too small a market. There are plenty of drug programs going after way-too-small markets because of all these wacky perverse incentives created by insurers and regulators to incentivize the creation of billion-dollar++ drugs for diseases virtually no one has.
I’ve been hearing for 10 years now on how “AI” will change this clinical rate. It’s now like the “This is the year of Linux” prediction meme. And like Derek has mentioned multiple times you can’t statiscialise biology. I don’t think many people get it so we have tech bros like us throw AI at data.
I've been hearing some form of it since I was an undergrad. I just submitted by packet for full professor. There have been some spectacular strides, but we've been through multiple rounds of "Stand aside, the computers will take it from here."
People saying this stuff aren't in the game. ML has been used in clinical research for basically as long as various ML methods have been around. Basically a century now in other words. AI leaders are trying to get money from people who don't understand the methodology such as investors or trump administration officials. Imagine trying to convince people who actually work day in day out with classic ml and deep learning to use your llm goobly goop bot.
Why does he conclude there is no alternative? He already argued this would be strange in other disciplines. So reform it.
If you can think of a way that you can have a much higher chance than 9% of success in getting a useful treatment to market, you can make a boatload of money. Go ahead, reform it.
Not knowing of a way is inequivalent to inexistence of any way. This is basic and apparently some bloggers are bad writers and or political conservatives of the type "there is no alternative".
The alternative is "have a complete mechanistic understanding of biology." People are definitely working on it, but we are nowhere close to as complete an understanding as we need. The evidence for precisely this claim is in the failure rate of clinical trials, into which we encode our best-known understanding of biology and we still end up being wrong 90% of the time. And that's not including all the pre-clinical candidates, which drive our wrongness rate (i.e. knowledge gap) to 99.99999%+
You can go make literally a trillion dollars if you can come up with an alternative.
Irrelevant hypothetical.
How? He’s saying that he can’t think of any good ones, and in my experience that’s true of most informed people. This is one of those hard problems that needs solutions, and so far the present system is the best anyone has managed.
I don't care what he's "saying that" which is just your biased interpretation. His essay does not warrant the final concluding sentence which would require a different essay thesis. It's just basic high school reading and writing skills.
Again, anyone who cracks a decent success rate can basically print money. Unfortunately, despite lots of efforts in this area, we have not been able to get the clinical success rate up.
It's pretty low effort to jeer from the outside and say "do better" for an industry that's been trying so many approaches to do so, without any ideas yourself.
The techniques have gotten a lot more sophisticated, but unfortunately a lot of the low hanging, "easy" fruit have been used up. Biology is hard.
It's not for lack of trying. People are definitely trying to find better ways of finding stronger candidates earlier. People are trying to gain a more precise understanding of disease mechanisms.
There are breakthroughs that I could have never imagined such as mRNA vaccines.
I would have to reread the piece carefully but I don't think "lack of trying by Biopharma" is consistent with the author's main argument. Because he is still being somewhat critical of the way it is currently done (why he cites the other industries). I simply disagree with his final sentence since it requires a different set of essay arguments, and also is subjective in the way say a political centrist might say "there is no alternative to fossil fuels" etc.
Most other industries barely innovate. Most industries are just buying off the shelf components and putting them together into larger widgets. Cars, computers, airplanes, that's the business. The components might get more efficient or performant but they generally do the thing they always did. If you squint it is basically the same car/plane/laptop as x years ago. They worked out the optimal car/plane/laptop shape a while ago and now everything sold is exactly that shape and will be for probably the rest of both of our lives and then some, in those industries.
Life science on the other hand is actually on a bleeding edge. Something actually revolutionary might emerge tomorrow after a grad student has an acid trip tonight.
Umm restaurants ARE like that, why do you think there’s so many burger and pizza joints?