There‘s an idea somewhere inside every Fortune 500 business right now that represents tens of millions of dollars of revenue growth or cost savings. One idea buried in brainstorms at Microsoft led to $120 million of annual revenue growth. “Bing’s revenue grew every year for several years now because of ideas like this, and because we didn’t ship bad stuff. Not shipping is indirectly cost saving, but many miss this point,” said Ronny Kohavi, Distinguished Engineer at Microsoft and former Director, Data Mining and Personalization at Amazon.

A single, simple idea to change the way ad titles were displayed on the Bing search engine was relegated to the bottom of a long list of ideas to grow revenue for Microsoft. In an account published in The Surprising Power of Online Experiments by the Harvard Business Review, Kohavi and Stefan Thomke gave more color to the story:

“Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a ‘too good to be true’ alert.”

The idea to move ad text to the title line and make it longer increased Bing’s revenue by $120M annually. The key was that they treated the idea as an experiment and just made it happen along with hundreds more promising ideas. They tested all of these ideas to prove which ones actually worked to improve key performance indicators.

His change to the search engine triggered an alert that something was wrong with revenue. The problem was that Bing was making too much money.

During the 2018 NextHealth Executive Advisory Council meeting, Kohavi shared his lessons learned how to build a culture of experimentation. These learnings offer pragmatic advice for healthcare organizations looking to uncover both big and small wins that radically transform their business.

Agree on a good Overall Evaluation Criterion (OEC)

In the short-term, it was easy to make money by showing more ads on Bing. Yet Kohavi and his team knew it increased abandonment rates which negatively impacted long-term revenue. The key was to balance measurement and optimization for short-term impact that didn’t compromise long-term goals. So, the OEC defined short-term metrics that predicted long-term value and were hard to game.

Action item for healthcare executives: To combat silos and opinion-based decision making, secure agreement on critical metrics to allow your team to focus on ways to impact that metric for long term business benefit.

Most ideas fail

At Microsoft, only one-third of its ideas have statistically significant positive impact on key metrics. Only one out of 5,000 experiments improves Bing’s most important metric of sessions per user.

Action item for healthcare executives: Since it takes a high volume of tests to find the ones that work, experiment often and make it easy to experiment at scale. For further reading, Kohavi recommends work by Michael Schrage at MIT who has coined the term “Iterative Capital.”  

Small changes can have big impact

It’s important to test often and in high volume because more tests mean more opportunities to find improvements. Site links in Bing advertisements add $50M annually to Microsoft’s business. Different font text color in the Bing experience delivered over $10M annually and credit card offers on Amazon’s shopping cart add tens of millions of dollars annually.

Action item for healthcare executives: There are programs and insights in health plans with value in the hundreds of millions of dollars waiting to be discovered. Start testing now to be able to increase the volume of tests over time.