Preprint: What 21 Clinical Trials Taught Us About Predicting the Next One (Part 1)
Every clinical development team knows the question that haunts a go/no-go meeting: what if we're wrong, and we don't find out for three years and after millions of dollars are spent? That question is what our team set out to chip away at in a new validation study, now available as a preprint.
The study, developed with Sanofi and institutions like Mount Sinai, examines the effectiveness of a QuantHealth modeling framework built to simulate ulcerative colitis (UC) trials. Rather than lead with headline numbers, it is worth walking through how the validation was designed, because the design is what makes the numbers mean anything.
The Setup
The model combines two data sources that don't usually talk to each other:
- Real-world data: featuring data from nearly 483,000 real UC patients, drawn from insurance claims and EHRs, capturing how patients respond to treatment outside the artificial conditions of a trial.
- A biomedical knowledge graph: approximately 100,000 biological entities connected by more than 5 million relationships, encoding drug mechanisms, targets, and pathways.
Why the Validation Was Designed This Way
Most predictive models in this space get validated against one or two trials, which says little about generalizability. This effort set a higher bar: 21 real randomized controlled trials, 13 drugs, phases 2 and 3, spanning 2007 to 2022, recreating each trial's actual eligibility criteria, endpoint definition, measurement timepoint, and arm sizes, then scoring the model’s predictions against the published results.
The Headline Results
This validation asked a harder question than a single accuracy percentage. For each trial, the model had to get both the direction of the treatment effect and its statistical significance right. That is the same bar a go/no-go decision must clear.
- 16 of 21 trials were correctly predicted: right direction, right statistical significance, resulting in an overall F1 score of 0.80.
- When the model predicted a trial would succeed, it was right 10 times out of 11. For a go/no-go conversation that is the number that matters, because the expensive mistake is greenlighting a trial that fails.
- Of the 5 misses, 4 still pointed in the right direction. Three of those four were the model being conservative: it predicted a real benefit but stopped short of statistical significance. One went the other way, predicting significance where the trial found none. Only one trial (VARSITY, the longest follow-up in the set at 52 weeks) reversed direction entirely.
- VARSITY is worth understanding rather than glossing over. It is the only trial in the set with a 52-week endpoint, in a field where the rest measured between 6 and 14 weeks, and the error traced to the control arm rather than the treatment arm. Longer follow-up in real-world data carries more informative censoring and more uncontrolled concomitant treatment. In HIBISCUS-1, which used the same comparator at week 10, the model matched the trial exactly. The practical read: this model is validated for induction-length endpoints, and that is where it should be used.
An F1 score of 0.80 across 21 trials, on its own, is a solid result. But it's not the most interesting number in this study. Six of the 21 trials involved agents the model had no real-world prescribing data to learn from, because those drugs were held out of the training data and represented only through the knowledge graph. The model got all six right, its best performance in the set.
That result deserves more than a mention, so we're giving it the full second post: why it happened, what it means for assessing first-in-class or repurposed drugs, and where the caveats are.
The full preprint is available here.
If you are weighing whether simulation belongs in your own go/no-go process, the honest test is simple: pick a trial you have already read out, and let us predict it without showing us the answer. That is the best use of an hour after a demo.