In AI-for-science, every demo shows the wins. The models that converged, the plots that fit, the benchmarks that beat the baseline. The failures are quietly dropped — or never run at all. We took the opposite stance: we built a public verification suite of 62 scenarios, and we publish the verdict for every single one, including the 5 that failed.
Every scenario is documented with a 10-chapter Verification & Validation (V&V) report aligned to ASME V&V 10-2019 and FDA CM&S expectations: problem statement, model description, code & data provenance, numerical accuracy, uncertainty quantification, sensitivity, calibration, limitations, and a reproducibility manifest. Each report ends with a single machine-checkable verdict.
The verdict is not a vibe. It is computed from out-of-sample R² and a one-sided prediction-interval coverage:
R² ≥ 0.90 and coverage ≥ 0.90R² ≥ 0.70 and coverage ≥ 0.80All five REFUTED scenarios share one root cause: they are inverse parameter-identification problems in microbial kinetics, where a Gaussian-Process surrogate is asked to recover parameters it was never given. The reported fits are poor (R² between 0.12 and 0.56), and we keep them visible rather than silently removing them.
2-D Monod kinetics: recover growth rate and half-saturation from sparse design-of-experiments samples. The forward model is smooth, but inverting it from 50 noisy points is ill-conditioned; the GP latches onto the wrong ridge.
Maximum-likelihood Monod fitting in 1-D. Identifiability collapses at low substrate — the classic flat-likelihood problem. Coverage stays high (0.93) only because the uncertainty band is honest enough to admit "I don't know."
Pirt relationship between growth and product formation. Two coupled rates, one observable; the surrogate cannot separate them from sparse data.
Two-strain competition. The losing strain is suppressed below the measurement floor, so its dynamics are unobservable — a data-limits problem, not a model bug.
Cross-species (E. coli / yeast) competition. The interaction term is weakly excited by the training design, leaving the GP with almost no signal to learn.
None of these are coding errors. They are known-hard inverse problems where sparse observations simply do not contain the information. Hiding them would have inflated our pass rate to 100% and made the other 53 numbers meaningless.
A high pass rate is easy to fake. We adopted three constraints specifically to make cheating harder:
The bottleneck in AI-driven research has shifted. Generating candidate results is now cheap; deciding whether to trust them is the new constraint. Credibility is not a bigger model — it is quantified uncertainty plus the ability for an independent party to re-run your verification.
Most tools treat a failed run as something to discard. We treat REFUTED as a first-class output: it tells a researcher exactly where the surrogate cannot be trusted, which is the most actionable signal a validation layer can give. That is the product — not "we are always right," but "here is precisely where we are not, and why."
In September 2026, GPT-6 Astra topped Insilico Medicine's DDD antibody-developability benchmark with no plugin and no fine-tuning — a clear signal that general research agents can now generate scientific results. But the same reviews are blunt: these agents are "prone to plausible-but-wrong outputs, weak guardrails," and every deployment needs quantified uncertainty plus an independent checkpoint. Our 62-scenario suite is that checkpoint. It does not care whose model produced the prediction — feed it any model's points and it returns an honest R² / coverage / calibration verdict, and a hard BLOCK when the prediction is not trustworthy. Generation got cheap; the scarce asset is the layer that can say no.
① We publish all 62 verdicts, including 5 REFUTED — honesty is the feature, not the marketing.
② The 5 failures are inverse-parameter-identification limits of sparse data, not bugs; they define the trustworthy boundary.
③ In AI4S, the scarce asset is a verification layer you can audit, not another model that claims to be correct.
Are the 62 scenarios real experiments?
No — and we are explicit about it. They are physics-informed virtual experiments validated against published analytic solutions and literature benchmarks (e.g. Verhulst logistic growth, Monod kinetics). They are a verification asset, not a substitute for wet-lab measurement. We have not yet run them against our own bench data.
Why keep the REFUTED models on display instead of fixing them?
Because the failure mode is informational, not procedural. Throwing more compute at an ill-conditioned inverse problem does not recover unmeasured parameters. The honest move is to mark the boundary and let users know where not to rely on the surrogate.
Can I see the reports?
Yes. Every scenario has a 10-chapter HTML/JSON/DOCX report and the full suite is indexed at swarmlabs.tools/report-gallery. The scenario definitions and the REFUTED root-cause analysis are open-sourced at github.com/lm203688/swarmlabs-benchmarks.
Factual verification: all numbers in this article are taken from the live report index at swarmlabs.tools/v3/report-index on 2026-09-12/13 (PASS 53 / MARGINAL 4 / REFUTED 5 / ERROR 0). The 5 REFUTED figures are the measured out-of-sample R² values from that index. SwarmLabs does not present these virtual-experiment results as wet-lab measurements.