ARTICLE
Does experimentation maturity drive business growth?
Experimentation helps businesses make better decisions before they commit time, budget or customer trust. The evidence is strongest when experimentation is treated as decision infrastructure, not a one-off optimisation tactic.
TopicsMaturityDecision qualityExperimentation practice
Introduction
The question is not whether every A/B test creates growth. It is whether businesses make better decisions when they test important ideas with real customers before scaling them.
Research on data-driven decision making shows a link with stronger business performance, and many leading digital companies use experimentation to guide product, customer experience and growth decisions.
What the evidence points to
- Data-driven companies perform better. Research links data-driven decision-making with higher output and productivity. [1]
- Experiments show cause and effect. Controlled experiments help teams understand whether a change caused customer behaviour to shift. [2]
- Digital leaders test at scale. Google, Microsoft, Amazon, Booking.com and Netflix use experimentation to guide product and growth decisions. [3]
Why data-driven decision-making matters
Research on data-driven decision-making by Erik Brynjolfsson, Lorin Hitt and Heekyung Kim studied 179 large publicly traded companies. They found that companies using data-driven decision-making had output and productivity 5-6% higher than expected, after allowing for other investments and technology use. They also found links with asset utilisation, return on equity and market value.[1]
This study supported that companies that use data and analytics to make decisions perform better than those that rely more heavily on opinion, hierarchy or instinct.
Experimentation is one practical way to make data-driven decision-making real. It changes the question from “What do we think will work?” to “What did customers actually do when we tested it?”
That shift is important. It moves teams away from guessing and closer to learning from real customer behaviour.
Why experimentation is different from ordinary analytics
Analytics can show what happened. Experimentation helps show whether a specific change caused what happened.
For example, analytics might show that conversion increased after a new homepage launched. But that does not always prove the new homepage caused the increase. The result could have been influenced by seasonality, marketing activity, competitor behaviour, pricing, weather or many other factors.
A controlled experiment creates a fairer comparison. Some users see the current experience. Others see the changed experience. This makes it easier to understand whether the change itself made a difference.
Researchers Ron Kohavi and Roger Longbotham describe online controlled experiments, including A/B tests, as a way to evaluate ideas quickly with real users. Unlike many data analysis methods that show correlation, controlled experiments can help establish cause and effect with a high level of confidence.[2]
For leaders, this is the strategic value of experimentation. It reduces guesswork in decisions that would otherwise be based on assumptions.
Leading digital companies invest heavily in experimentation
Large digital companies do not treat experimentation as a small optimisation tactic.
A 2022 review of online controlled experimentation states that companies including Airbnb, Alibaba, Amazon, Booking, Alphabet’s Google, LinkedIn, Meta’s Facebook, Microsoft, Netflix and Uber have invested tremendous resources in online controlled experiments. They use experiments to understand how innovation affects customers and business outcomes.[3]
Google is one of the clearest public examples. Google says all possible changes to Search go through a careful evaluation process before launch. In 2023, Google ran more than 700,000 experiments, which led to more than 4,000 improvements to Search. This included 16,871 live traffic experiments, 719,326 search quality tests and 124,942 side-by-side experiments.
Google explains its launch standard clearly: “If we can’t show that a change actually makes things better for people, we don’t launch it.”[4]
Microsoft has also invested deeply in experimentation. Microsoft Research describes its Experimentation Platform, known as ExP, as a system for trustworthy experimentation. Its role is to help teams move from an idea to an insight, test hypotheses, measure real-world impact and safely improve products used by billions of people.[5]
The commercial value can be significant. Harvard Business Review reported that a Microsoft Bing experiment involving ad headlines increased revenue by 12%, worth more than US$100 million annually in the United States alone, without harming key user-experience metrics.[6]
The important lesson is not just the size of the result. It is that the idea had been underprioritised until an experiment showed its value.
Booking.com is another strong example. A paper on Booking.com’s experimentation approach describes more than ten years of evidence-based product development using online experiments. It highlights decentralised decision-making, shared records of successful and unsuccessful experiments, data-quality checks and safeguards that allow many people to own experiments.[7]
Amazon’s published research shows experimentation being used for complex business decisions, including pricing. Amazon Science describes Amazon Pricing Labs, which runs online A/B experiments to evaluate new pricing policies while accounting for constraints such as nondiscriminatory pricing, spillover effects and demand trends.[8]
Netflix is also clear that experimentation is part of how the company learns and decides. Netflix says A/B test results and other causal inference methods are generally expected to inform product decisions. It has invested in culture, people, infrastructure and internal education to make A/B testing easier for teams across the company to use.[9]
How leading companies view experimentation success
The most advanced companies do not define experimentation success only as more winning tests.
Google frames experimentation success around usefulness and launch quality. It runs experiments to decide whether changes make Search better for users. If a change cannot show improvement, Google says it does not launch it.[4]
Microsoft frames experimentation as a way to support innovation, real-world measurement and safe iteration. Its ExP platform helps product, engineering and data teams test ideas and measure impact across large-scale systems.[5]
Netflix has also written about return-aware experimentation. In simple terms, this means experiments are not only scientific tests. They are decision tools that help the business choose options that are more likely to improve important metrics over time.
Netflix researchers used 123 historical A/B tests to evaluate decision rules. They estimated that one new rule would increase cumulative returns to a north-star metric by 33%, which led Netflix to adopt the rule.[10]
The goal is not simply to run more tests. The goal is to make better decisions, use resources more effectively, avoid poor launches and identify changes that create customer and business value.
What this means for experimentation maturity
Experimentation maturity is best understood as a business capability, not a target for test volume.
A mature experimentation program is not just a team with an A/B testing tool. It is a way of working that helps a business move from opinion-led decisions to evidence-led decisions.
In practice this means:
- Clear hypotheses
- Reliable data
- Agreed success metrics
- Appropriate statistical methods
- Clear decision rules
- Shared learning across teams
- Leadership support
A mature experimentation program improves the conditions for growth by increasing decision quality, speeding up learning, reducing launch risk and helping useful gains compound over time.
Experimentation helps businesses grow by improving how they make decisions. It gives teams a way to test strategic assumptions, validate customer behaviour, reduce costly mistakes and learn faster than competitors that rely mostly on opinion or delayed performance reports.
What leaders should measure
To understand whether experimentation is helping the business grow, leaders should measure more than test volume.
Running more tests can be useful, but only if the tests are meaningful, well-designed and connected to real business or customer questions.
A better way to measure experimentation value
- The percentage of roadmap decisions informed by experiments
- The rate at which successful experiments are shipped permanently
- The cost of launches avoided because evidence showed they were unlikely to work
- The time from customer insight to tested decision
- How often learnings are reused across teams
- Impact on customer and commercial metrics
- The quality of decisions made after inconclusive or negative tests
This changes experimentation from a conversion-rate activity into a decision-quality system.
The bottom line
Experimentation is a way to make better business decisions.
Data-driven decision making is linked with stronger business performance. Controlled experiments are a powerful way to understand cause and effect. Many of the world’s leading digital companies have made experimentation central to how they improve products, customer experiences and business outcomes.
The companies that gain the most from experimentation are not simply the ones that run the most tests. They are the ones that use experimentation to learn faster, reduce risk and make better decisions again and again.
Questions
FAQs
- Does experimentation cause business growth?
- Experimentation’s value is that it improves decision quality. It helps businesses identify, validate and scale better ideas while reducing the risk of investing in ideas that do not work.
- Is there evidence that data-driven companies perform better?
- Yes. Brynjolfsson, Hitt and Kim studied 179 large publicly traded companies and found that companies using data-driven decision making had output and productivity 5-6% higher than expected. They also found links with asset utilisation, return on equity and market value.
- Why do leading companies run so many experiments?
- Leading digital companies use experiments to evaluate ideas with real customers before making broader launch decisions. Google, for example, ran more than 700,000 Search experiments in 2023, resulting in more than 4,000 improvements.
- Is A/B testing only useful for conversion rate optimisation?
- No. A/B testing is often used in conversion rate optimisation, but leading companies use experimentation more broadly. They use it for product development, search quality, pricing, personalisation, advertising, customer experience and platform decisions.
- What is the main value of experimentation maturity?
- Maturity is not simply running more tests. It is having a repeatable way to make better decisions, learn faster, reduce launch risk and build customer knowledge that becomes more useful over time.
Sources and further reading
Academic and research evidence
Brynjolfsson, Hitt and Kim, Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?
Research on 179 publicly traded firms showing an association between data-driven decision-making and stronger productivity, asset utilisation, return on equity and market value.
Kohavi and Longbotham, Online Controlled Experiments and A/B Tests
Explains why online controlled experiments help teams evaluate ideas quickly and establish cause and effect with higher confidence than correlation-based analysis alone.
Larsen, Stallrich, Sengupta, Deng, Kohavi and Stevens, Statistical Challenges in Online Controlled Experiments
Reviews online controlled experimentation and states that major digital companies, including Airbnb, Alibaba, Amazon, Booking, Google, LinkedIn, Meta, Microsoft, Netflix and Uber, have invested heavily in experimentation.
Company experimentation examples
Google Search, Improving Search with rigorous testing
Describes Google’s Search evaluation process and reports that in 2023 Google ran more than 700,000 experiments, leading to more than 4,000 Search improvements.
Microsoft Research, Experimentation Platform (ExP)
Describes Microsoft’s ExP platform as a system for trustworthy experimentation that helps teams validate hypotheses, measure real-world impact and safely iterate on products used by billions of people.
Kaufman, Pitchforth and Vermeer, Democratizing online controlled experiments at Booking.com
Describes Booking.com’s decade-plus use of online experiments, including decentralised decision-making, knowledge sharing, data-quality monitoring and safeguards for broad experiment ownership.
Amazon Science, The science of price experiments in the Amazon Store
Explains how Amazon Pricing Labs uses online A/B experiments to evaluate pricing policies while accounting for constraints such as nondiscriminatory pricing and spillover effects.
Netflix Technology Blog, Netflix: A Culture of Learning
Describes Netflix’s experimentation culture, including broad buy-in for A/B testing, internal education and the expectation that A/B test results or causal inference should inform product decisions where possible.
Commercial impact and decision-making
Kohavi and Thomke, The Surprising Power of Online Experiments
Harvard Business Review article describing the Microsoft Bing ad-headline experiment that increased revenue by 12%, worth more than US$100 million annually in the United States alone.
Netflix Technology Blog, Return-Aware Experimentation, and Chou et al., Evaluating Decision Rules Across Many Weak Experiments
Explains return-aware experimentation and supports the example of Netflix researchers using 123 historical A/B tests to evaluate decision rules, estimating a 33% increase in cumulative returns to a north-star metric. See also: https://arxiv.org/abs/2502.08763
Harvard Business School Executive Education, The Critical Role of Leadership in Building a Culture of Experimentation
Discusses experimentation culture and cites Baker Research Services comparing companies with strong experimentation infrastructure and culture against the S&P 500 over a ten-year period.