A Data‑Driven Evaluation of Optimization Techniques in Cell Culture Media
Media formulation can account for up to 95% of production costs in cellular agriculture, yet deciding what to try next remains slow, manual, and expensive. This project evaluates how far machine learning—especially active learning and Bayesian optimization—can go in reducing experimental waste and accelerating discovery, using benchmarks grounded in real datasets and realistic constraints.
The Problem: Why Traditional Methods Fall Short
For decades, researchers have relied on Design of Experiments (DOE) methods to systematically explore experimental spaces. While DOE brings valuable structure and statistical rigor, it has a critical limitation: it's non-adaptive. Once you design your experimental plan, you're locked in—even if your first few results scream "look over here!" at a promising region you didn't anticipate.
In the complex world of cell culture media optimization, this inflexibility comes at a steep cost:
- High dimensionality: Media formulations can involve dozens of components, each with its own concentration range
- Noise and variability: Biological systems are inherently noisy, making it hard to distinguish signal from noise
- Expensive experiments: Each test run costs time, materials, and resources
- Fixed plans: Traditional DOE commits you to a predetermined path, wasting opportunities to learn and adapt
The result? You often need many more experiments than necessary to find optimal conditions. What if we could be smarter about which experiments to run next, learning and adapting as we go?
The Innovation: Two Benchmarks that Mirror Real Labs
We've created two complementary benchmarks that capture the essence of real laboratory campaigns. Each tests different optimization skills and scenarios you'll encounter when developing cell culture media:
Hide-the-Label (HTL)
The Scenario: Imagine you have a library of 100 different media formulations sitting on your shelf. You know they've all been tested before, but the results are hidden from you. You can only afford to test 20 of them. Which ones do you choose?
What it tests: Your ability to strategically select from a finite pool of candidates, balancing exploration (trying diverse options) with exploitation (focusing on promising regions). This mirrors real scenarios where you have a candidate library and need to efficiently screen it.
Goal: Reach your target performance as quickly as possible, or maximize the best result within your budget.
Open Race (OR)
The Scenario: You're starting from scratch in a continuous experimental space. Each ingredient can be varied smoothly within certain ranges. Where do you explore? How do you balance trying new regions versus refining promising ones?
What it tests: Your ability to navigate continuous optimization spaces, propose intelligent batches of experiments, and efficiently converge toward optimal formulations. This is like designing a media recipe from the ground up.
Goal: Track your best-so-far performance over time and reach the optimum with minimum experiments.
Both benchmarks come in two difficulty levels:
- Regular Mode: Uses smooth Gaussian Process (GP) surrogates with moderate noise—ideal for understanding baseline performance
- Hard Mode: Introduces real-world complexities like heteroscedastic noise (varying uncertainty), discontinuities, and multiple local optima—testing robustness under challenging conditions
How Hide-the-Label Works: Think of it Like a Game
Imagine you're playing a card game where 100 cards are laid face-down on a table. Each card has a score written on it (representing media performance), but you can't see them. You have a budget to flip over only 20 cards. Your goal? Find the highest-scoring card as quickly as possible!
This is exactly what the Hide-the-Label benchmark does. It takes real experimental data and "hides" most of the results. Different optimization strategies then compete to see which ones can most efficiently reveal the best candidates. Smart strategies will:
- Start with diverse exploration to understand the overall landscape
- Focus on promising regions once patterns emerge
- Balance risk and reward by choosing candidates with high predicted value and high uncertainty
- Learn from each reveal to make smarter next choices
How Open Race Works: Navigating Uncharted Territory
Unlike Hide-the-Label where you're choosing from existing options, Open Race is like exploring a vast, continuous landscape searching for the highest peak. You don't have a map, and you can only sample a limited number of locations. Each measurement tells you the "elevation" (performance) at that spot.
The challenge is to develop a smart exploration strategy that:
- Builds an internal model of the landscape based on your observations
- Proposes new experiments in batches to efficiently use resources
- Tracks improvement over time to measure convergence speed
- Adapts the search strategy as you learn more about the space
This mirrors the real-world scenario of formulating media from scratch, where you can mix ingredients at any concentration within safe ranges and need to find the optimal combination efficiently.
Key Insights: What We Discovered
After running thousands of benchmark trials across multiple datasets and difficulty levels, we uncovered some surprising and actionable insights. Each claim below is qualified with the specific benchmark configuration it refers to—use the Interactive Playground to reproduce any result by selecting the matching parameters. In the playground, optimizers are grouped into three categories (Classical DOE → Direct Heuristic → Surrogate-Assisted) so the overall trend is immediately visible in the chart layout.
1. BO_GP_EI and SBO_GP_PV Lead the Pack
GP-based Bayesian Optimization with Expected Improvement (BO_GP_EI) and Smart BO with Predictive Variance (SBO_GP_PV) emerged as strong performers across both benchmarks under Regular Mode (GP surrogate, low noise). Under this configuration, BO_GP_EI found near-optimal candidates in as few as ~4–10 steps on the Hide-the-Label benchmark (95% hidden, Batch 1), while random selection required far more. In Open Race under the same Regular Mode, both GP-based methods achieved the highest final performance values. Note: this advantage is most pronounced in Regular Mode; Hard Mode results vary by dataset—see the playground for dataset-specific comparisons.
Why it matters: These methods dramatically reduce experimental costs and timelines by intelligently balancing exploration and exploitation, making them ideal for expensive biological experiments.
2. BO Methods Dominate Traditional Approaches
Across experiments in both Regular and Hard Mode, Bayesian Optimization methods (BO_GP_EI, SBO_GP_PV) consistently outperformed traditional Design of Experiments (DOE) approaches such as Full Factorial, Latin Hypercube, and Central Composite designs. The BO advantage is strongest under Regular Mode / Hide-the-Label / Batch 1. In Hard Mode or larger batch sizes the gap narrows on some datasets, which is why we recommend checking the playground under your specific conditions.
The lesson: Traditional DOE methods still have value for structured exploration, but when sample efficiency matters most, BO-based active learning is the superior choice. Random sampling performs dramatically worse than all systematic approaches.
3. Statistical Rigor Prevents False Claims
We didn't just compare average performance—we used permutation testing, confidence intervals, and statistical power analysis to ensure our conclusions were solid. Many methods that looked good in single runs didn't hold up under rigorous statistical scrutiny. All results in the playground reflect averaged performance over multiple competitions and (for Hide-the-Label) multiple synthetic datasets, not single runs.
Critical insight: Always validate claims with proper statistics. What looks like a win might just be random noise!
4. Hard Mode Reveals True Robustness
Our Hard Mode uses a Random Forest surrogate (instead of GP) with added homoscedastic noise (σ = 0.1), making the black-box function harder to model. Under these conditions, the relative ranking of optimizers can shift—methods that excelled under Regular Mode may not hold the same lead. Use the playground's Difficulty Mode toggle to see how each optimizer's ranking changes between Regular and Hard configurations.
Practical advice: Always benchmark under conditions that match your real experimental challenges—not just idealized scenarios. If your lab data is noisy or non-Gaussian, prefer Hard Mode results when choosing an optimizer.
Practical Roadmap: How to Apply This in Your Lab
Ready to bring these insights to your own media optimization projects? Here's a step-by-step guide to get started:
Step 1: Start with the Best of Both Worlds
You don't have to choose between Bayesian Optimization and traditional DOE—combine their strengths:
- Use GP-based BO for adaptive experiment selection (the "what to try next" decisions)
- Keep DOE's discipline for experimental design (proper factor bounds, randomization, controls, and replication)
- This hybrid approach gives you both intelligence and rigor
Step 2: Test Before You Invest
Before committing to expensive wet-lab experiments:
- Run simulations using our benchmarks with datasets similar to yours
- Test different optimizers under both Regular and Hard modes
- Understand which strategies work best for your specific problem characteristics
- This "dry run" can save thousands in lab costs and weeks of time
Step 3: Use the Public Codebase
Everything you need is open source and ready to use:
- Reproduce our results: Run regular/hard × HTL/OR experiments out of the box
- Customize optimizers: Modify or add your own in
utils/optimizers.pyandutils/open_race_optimizers.py - Plug in your data: Add your datasets via
utils/datasets.py - Compare fairly: All methods are evaluated on the same benchmarks with the same metrics
Step 4: Measure What Actually Matters
Don't just look at final performance—track the full story:
- Steps-to-target (HTL): How quickly did you find a solution?
- Best-so-far trajectories (OR): How does performance improve over time?
- Confidence intervals: Are the differences statistically meaningful?
- Permutation tests: Could this result have happened by chance?
- These metrics tell you if an optimizer is reliably better, not just lucky once
The Bottom Line: Smarter Experiments, Better Results
The future of experimental biology isn't just about doing more experiments—it's about doing smarter experiments. With data-driven benchmarking and adaptive optimization, we can:
- Cut experimental waste by 50-90% using BO_GP_EI or SBO_GP_PV instead of random sampling
- Accelerate discovery timelines from months to weeks with adaptive experiment selection
- Make optimization decisions with statistical confidence, not gut feeling
- Outperform traditional DOE methods while maintaining experimental rigor
- Reduce costs dramatically by testing only what's most informative
"When the goal is fewer, more informative experiments—not just more experiments—adaptive optimization becomes a force multiplier for the lab."
Ready to Get Started?
Try our interactive playgrounds below to see how different optimization strategies compare!