Every clinical development team knows the question that haunts a go/no-go meeting: what if we're wrong, and we don't find out for three years and after millions of dollars are spent? That question is what our team set out to chip away at in a new validation study, now available as a preprint.
The study, developed with Sanofi and institutions like Mount Sinai, examines the effectiveness of a QuantHealth modeling framework built to simulate ulcerative colitis (UC) trials. Rather than lead with headline numbers, it is worth walking through how the validation was designed, because the design is what makes the numbers mean anything.
The Setup
The model combines two data sources that don't usually talk to each other:
Why the Validation Was Designed This Way
Most predictive models in this space get validated against one or two trials, which says little about generalizability. This effort set a higher bar: 21 real randomized controlled trials, 13 drugs, phases 2 and 3, spanning 2007 to 2022, recreating each trial's actual eligibility criteria, endpoint definition, measurement timepoint, and arm sizes, then scoring the model’s predictions against the published results.
The Headline Results
This validation asked a harder question than a single accuracy percentage. For each trial, the model had to get both the direction of the treatment effect and its statistical significance right. That is the same bar a go/no-go decision must clear.
An F1 score of 0.80 across 21 trials, on its own, is a solid result. But it's not the most interesting number in this study. Six of the 21 trials involved agents the model had no real-world prescribing data to learn from, because those drugs were held out of the training data and represented only through the knowledge graph. The model got all six right, its best performance in the set.
That result deserves more than a mention, so we're giving it the full second post: why it happened, what it means for assessing first-in-class or repurposed drugs, and where the caveats are.
The full preprint is available here.
If you are weighing whether simulation belongs in your own go/no-go process, the honest test is simple: pick a trial you have already read out, and let us predict it without showing us the answer. That is the best use of an hour after a demo.