By PAUL VON BÜNAU
Immense development costs and low success rates can make drug development look like a fool’s errand. But an algorithm that predicts the likelihood of failure with a 73% accuracy rate could change all that
The process of drug development is essential for bringing better medicines to patients. Yet, with an industry-wide probability of success estimated by some to be as low as ~10%,1 and development costs in the ballpark of 2 billion US$ or more2 one could argue that it would be better not to invest in drug development at all. Indeed, according to Eroom’s law (the opposite of Moore’s law, postulated by Scannell and colleagues in 2012), R&D productivity has exponentially decreased over the past 50 years.3
Failures are especially problematic during the later stages of development, which account for almost half of the total development costs.2 Unsuccessful trials not only deplete developer’s cash reserves; they also hurt patients, subjecting trial participants to ineffective – or, at worst, harmful – treatments. The resources wasted on failed trials could also be put to better use on more promising applications.
To optimise resource allocation, drug developers need to gauge the development risk before investing in a certain asset, technology or indication. The same is true for private investors, who provide an important cash stream for the biopharma sector and need to identify which options their money should be committed to.
Current decision-making models in biopharma are often driven by preferences of experienced executives and look to historical success rates and expert input for guidance. Predictive instruments may be a first step towards changing the overall approach in favour of a more objective, transparent, and perhaps even inclusive way of making such important decisions.
Looking to the past to predict the future
By the time a drug reaches late stage development, some risk-mitigating insights about the test drug should have already accumulated, and trials are less prone to fail than at earlier stages. The money at risk, however, is more significant, as trials grow not only larger – in many indications spanning thousands of patients – but also longer, sometimes taking several years.4
Before plunging into the costly endeavour of late-stage clinical trials, most drug developers try to assess the probability of success (POS) by some means. Their methods tend to be part science, part art – ranging from strict decision matrices to decision-makers’ gut feelings.
Alongside program-specific features, a key part of any assessment tends to be a benchmark of historical success rates – how many drugs made it from one clinical phase to the next and then finally onto the market. Although this gives some hints about the risk entailed in a certain therapy area, indication or technology, these averaged numbers neglect more specific information about the asset in question, such as:
- its development history (the strength and evidence of phase II trials, for example)
- the developing company’s experience with the technology
- and the indication and new developments within the indication that might ease the development path (such as clearer guidelines or trial endpoints)
Assessments are not immune to bias either; decision makers tend to look more favourably on their current program than historical failures – perhaps due to actual data they have gathered to de-risk their asset, but more often simply due to the costs already incurred.
How, then, can drug developers improve the estimation of risk for their late-stage programs?
Forward-looking prediction of phase III success
Algorithmic predictions have penetrated many areas of life – from logistics to entertainment, and from marketing to finance – often outperforming human decision making in terms of speed and accuracy.
Yet biopharma, despite its strong scientific focus, takes little advantage of predictive algorithms for portfolio decision making. Might this be due to the multi-faceted nature of factors that drive clinical trial success – including both hard indicators (such as phase II results) and soft indicators (such as organisational experience)?
We developed a novel Bayesian modelling approach that integrates different dimensions of available information to make a forward-looking prediction for a specific trial, with a focus on the oncology space5 (rather than analysing broad, backward-looking trends). To derive the parameters for our model, we integrated the rich pool of available prior information into a sophisticated scoring system, which captures both hard and soft indicators of a specific trial.
While the design of individual scores was driven by expert knowledge of colleagues with a long history of working for the biopharma industry, score weights were determined by the model and proved empirically highly predictive. The model was trained on publicly available data of completed phase III and phase II oncology studies initiated between 2003 and 2012, extracted from databases such as CT.gov, European Clinical Trial Database and PubMed.
Results were pretty impressive: our model calculates the distribution of POS, including credible intervals (a measure for displaying uncertainty of the prediction) for a phase III trial in oncology with a predictive performance of ~73%.
Two factors emerged as key drivers for the predictive performance of the model:
- the quality and strength of phase II data, indicating how well the preceding experiment was done and how good its outcome was
- the experience of the sponsor at the time of study initiation, indicating how well prepared the company is to carry out the experiment based on past experience, sponsor networks and the strength of its organisation
While those predictors appear intuitive, it is noteworthy that they are indication-agnostic factors, which perhaps strengthens the argument against focusing on indication-specific historical POS as a decision-making benchmark. It should be noted, however, that selecting only drugs that entered into phase III for the training data introduces a certain bias, deprioritising indications with low probability for a drug to reach late stage development.
Other forward-looking predictive models have introduced Bayesian and non-Bayesian methods to calculate trial and regulatory success based on phase II and phase III data with varying sizes of training data and success.6,7,8 Unlike these models, ours uses linear regression to reduce overfitting, which makes it easier to use. At the same time, parameters are calibrated in a Bayesian fashion, enabling the use of credible intervals and a posterior predictive distribution for the POS intervals. One of the great strengths of our model is the expertise-driven refinement of composite scores, which proved highly predictive.
Translating trial success predictions into portfolio decision making
While a predictive model that assesses the likelihood of phase III success after completion of phase II but before the start of phase III feels useful at first glance, the big question is how those predictions can be turned into better decision making for biopharma companies’ budget allocations.
Two application scenarios come to mind:
- ‘Picking the winners’ into phase III (big pharma)
The most obvious use of a predictive algorithm is in informing late-stage portfolio strategy decisions. In a real-world application, idalab’s algorithm was employed by a multinational pharma company that wanted to improve late-stage decision making for its oncology portfolio.9
- ‘Program optimisation for phase III success’ (big pharma and smaller biotechs)
Knowledge about which features of phase II trial designs drive phase III success could help set up phase II trials in a way that maximises the likelihood of phase III success
Better things to come?
In the future, our model could be expanded and customised to encompass additional features that would make it even more valuable for drug developers and biopharma investors. The model is currently validated for – and limited to – oncology, for which trial endpoints (such as response and survival rates or durations) are usually easy to interpret and compare. When translating the model to other therapeutic areas, such as central nervous system diseases, the more ambiguous nature of trial endpoints will have to be taken into account.
Another challenge that lies ahead is the translation of phase III success into regulatory success, going beyond the success probability of an individual trial to evaluate the likelihood of the regulatory agencies to approve a program, which might consist of a number of individual trials with different features.
Thus far, our model has been proven to be a highly effective predictive tool – one that could help portfolio managers and investors make important, cost-effective and potentially life-changing decisions.
1. Hay et al. 2014, Nature Biotechnology
Comprehensive survey of historical success rates in clinical trials and often quoted for estimating industry averages
2. DiMasi et al. 2016, Journal of Health Economics
Study from the Tufts University with historical benchmarking for drug development costs, often quoted for estimating industry averages
3. Low et al. 2012
Blog that describes the principle of ‘Eroom’s law’, a reversal of Moore’s law and indicator of exponential decrease in biopharma productivity
4. FDA overview of clinical development giving benchmarks on trial sizes and durations
5. Hegge et al. 2020
Article showing the setup, training and performance of our phase III prediction model
6. Schater et al. 2007 Value Health
Article describing a Bayesian network approach for predicting success and economic outcomes of trials based on pre-phase III data
7. DiMasi et al. 2015 Clin Pharmacol Ther
Article describing a scoring-logic based model for predicting regulatory success based on phase II data of oncology drugs
8. Lo et al. 2019, Harvard Data Science Review
Article describing a machine learing algorithm based approach for predicting regulatory success based on phase II and phase III data of oncology drugs
9. Idalab case study, showing the application of the algorithm for portfolio decision making in a multinational biopharma company