The Failure to Learn What’s Really Working in Healthcare Is Driving Innovation

An interview with Dr. Scott Halpern of the University of Pennsylvania on innovation in testing what works to change healthcare outcomes.

What’s driving innovation in testing and developing interventions to improve healthcare?

Dr. Halpern: We need to improve knowledge about what works and what doesn’t. To change health-related behaviors, we need to improve the evidence around the highest impact and most cost-effective interventions. We must engage those who are in positions to implement interventions to use the evidence wisely so that they’re choosing the best interventions to make a difference

It’s really important when it comes to serious illnesses. We know a lot about how people die in this country. We understand what their last months and years are like and how these experiences impact their friends and family. Yet, we know very little about how to actually improve on the current state of affairs.

Why? The problem is that most interventions haven’t been rigorously tested to determine whether they actually achieve their goals and interventions. The few that have been rigorously tested either haven’t worked very well, or have only shown promise in narrow settings.

What’s the problem with current practices?

Dr. Halpern: First and foremost, it’s a failure of academics who are developing interventions to effectively partner with health systems, insurers, and others who determine healthcare for the majority of Americans. By partnering with these organizations to rigorously test interventions, academics could better achieve their mission of improving the current state of healthcare.

It’s also a failure of health systems and insurers to learn what works and what doesn’t. Too often, organizations are overcome by the very understandable impulse to just do something. Yet a more prudent response might be to take a step back and take the time necessary to evaluate what works in a particular setting and then implement what works best.

How are you getting buy-in to changing the way we test and learn?

Dr. Halpern: We’ve been very fortunate to collaborate with some really innovative and effective health systems, employers, and insurers around the country.

Three principles of innovation drive that partnership. One is that we map our programs to health priorities that health systems, insurers, and employers face, and we are adaptable in how we apply tools of behavioral change to reach consumers.

Second, we know that these organizations are feeling pressured to show evidence of change quickly–whether or not that evidence is beyond a shadow of a doubt.

The days of proposing a 6-year randomized control trial before we’ve got any data are changing. We now can propose much shorter time horizons for seeing change. It really speaks to the virtues of pragmatic testing designs. We can get answers faster.

The third thing that we’ve learned is to go beyond randomizing at the level of an organization. Before, we might have worked at the hospital level. Now, we’ve innovated with what are called “stepped-wedge” designs. Instead of randomly assigning some hospitals to get the intervention while others don’t, all the hospitals may get the intervention, but we randomly assign the time at which they adopt it. We allow those units to serve as their own controls. We compare those who already were randomly assigned to those who are still waiting.

That’s generated a ton more buy-in than the older way of doing things where some people are going to be left out entirely.

Is emerging technology changing the way we test and learn in healthcare?

Dr. Halpern: There’s no question. There are a variety of advances on the technological front which are greatly expediting our ability to not only conduct research but to affect change.

On the research side, we have better ways of predicting who is likely to benefit from an intervention and to target our interventions according to people’s underlying potential benefit using machine learning algorithms. We also use natural language processing to go above and beyond what we can learn about individuals simply through discrete data fields.

We are seeing new technologies for monitoring vital signs and measuring symptoms remotely. We can feed those data back into a machine learning algorithm. In the future, we may be able to process those data and present it to nurses, doctors, and other clinicians in real-time. That has the potential to allow them to better serve patients before that small problem becomes a big one.

The Holy Grail is if we can use insights from behavioral change science to deliver interventions through smartphones and other technologies.

Three Ways for Healthcare Companies to Uncover Hidden Value Through a Culture of Experimentation

There‘s an idea somewhere inside every Fortune 500 business right now that represents tens of millions of dollars of revenue growth or cost savings. One idea buried in brainstorms at Microsoft led to $120 million of annual revenue growth. “Bing’s revenue grew every year for several years now because of ideas like this, and because we didn’t ship bad stuff. Not shipping is indirectly cost saving, but many miss this point,” said Ronny Kohavi, Distinguished Engineer at Microsoft and former Director, Data Mining and Personalization at Amazon.

A single, simple idea to change the way ad titles were displayed on the Bing search engine was relegated to the bottom of a long list of ideas to grow revenue for Microsoft. In an account published in The Surprising Power of Online Experiments by the Harvard Business Review, Kohavi and Stefan Thomke gave more color to the story:

“Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a ‘too good to be true’ alert.”

The idea to move ad text to the title line and make it longer increased Bing’s revenue by $120M annually. The key was that they treated the idea as an experiment and just made it happen along with hundreds more promising ideas. They tested all of these ideas to prove which ones actually worked to improve key performance indicators.

His change to the search engine triggered an alert that something was wrong with revenue. The problem was that Bing was making too much money.

During the 2018 NextHealth Executive Advisory Council meeting, Kohavi shared his lessons learned how to build a culture of experimentation. These learnings offer pragmatic advice for healthcare organizations looking to uncover both big and small wins that radically transform their business.

Agree on a good Overall Evaluation Criterion (OEC)

In the short-term, it was easy to make money by showing more ads on Bing. Yet Kohavi and his team knew it increased abandonment rates which negatively impacted long-term revenue. The key was to balance measurement and optimization for short-term impact that didn’t compromise long-term goals. So, the OEC defined short-term metrics that predicted long-term value and were hard to game.

Action item for healthcare executives: To combat silos and opinion-based decision making, secure agreement on critical metrics to allow your team to focus on ways to impact that metric for long term business benefit.

Most ideas fail

At Microsoft, only one-third of its ideas have statistically significant positive impact on key metrics. Only one out of 5,000 experiments improves Bing’s most important metric of sessions per user.

Action item for healthcare executives: Since it takes a high volume of tests to find the ones that work, experiment often and make it easy to experiment at scale. For further reading, Kohavi recommends work by Michael Schrage at MIT who has coined the term “Iterative Capital.”  

Small changes can have big impact

It’s important to test often and in high volume because more tests mean more opportunities to find improvements. Site links in Bing advertisements add $50M annually to Microsoft’s business. Different font text color in the Bing experience delivered over $10M annually and credit card offers on Amazon’s shopping cart add tens of millions of dollars annually.

Action item for healthcare executives: There are programs and insights in health plans with value in the hundreds of millions of dollars waiting to be discovered. Start testing now to be able to increase the volume of tests over time.

terview with Dr. Steven Udvarhelyi, President and CEO of Blue Cross and Blue Shield of Louisiana

10 Lessons Learned in Driving New Healthcare Outcomes through Analytics with Dr. Steven Udvarhelyi

Highlights from an interview with Dr. Steven Udvarhelyi, President and CEO of Blue Cross and Blue Shield of Louisiana.

‘Fail fast’ isn’t just a mantra for high-tech companies. It’s how innovative health plans are driving better results, too. And that takes a culture that embraces analytics as a way to make better decisions.

Dr. Steven Udvarhelyi, President and CEO of Blue Cross and Blue Shield of Louisiana, shared his insights into building a culture of analytics with Eric Grossman, CEO of NextHealth Technologies at the 2018 Health Evolution Summit. Here are the 10 lessons he’s applying as he leads his organization to top results.


Lesson #1: Teach people to ask the right questions

In a lot of planning, you ask your team to build a financial plan without requiring an analytics plan for every initiative. Yet, it’s important to build a culture of analytics where people ask, “how do you know it’s going to work?”

People need to think about collecting data and information to make a good decision from the start. The evolution is to get people to stop saying, “I need a database or report.” Instead, we want them to come to a center of excellence to say: “I have a business problem; I need to understand how to build a business plan; I need to reach a population of members; I want to optimize a marketing campaign.” Very often, this means you have to change operations to collect information that’s not being captured today.


Lesson #2: Bake analytics right into the business plan and operations

At Blue Cross and Blue Shield of Louisiana, we won’t fund any work without an analytics plan. And, we’re doing that enterprise-wide, so that we’re not working in a siloed fashion. We’re working to try new things and fail fast, at scale. For example, we staffed a care management area with an analytics team recently. Now, they can test within eight weeks whether something is working or not. Then, they can decide whether to scale. Organizationally, it’s a way to have a high level of confidence that you’ll know what’s going to work and why. It’s a way to see what’s going to make a breakthrough in results.


Lesson #3: Get to one set of numbers for the truth

When working with analytics, the way to avoid tension between an analytics team and actuaries is to make everyone a part of the effort. There’s one set of numbers that we all use. How it works is that the analytics team collaborates with actuaries to estimate the impact of a program—such as determining the impact of a vendor that we’re evaluating. That’s how we eliminate churning about whose numbers are right.


Lesson #4: Be open to using non-traditional data sources

We’re finding that you have to look at non-traditional data sources. In Louisiana, we have a lot of people on Medicaid. It turns out that the number one variable that helps predict adverse health events is not in the claims system. It’s the credit score.

The FICO credit score is a powerful predictor of health outcomes. It lets you know who is vulnerable financially. When you’re vulnerable financially, as hourly employees often are, you tend not to have time to take off of work to visit a doctor. You can’t afford the loss of wages. You also tend to struggle with out-of-pocket expenses. Cost becomes a big factor in deciding to see a doctor. So, we experimented with eliminating cost-sharing to see if it changed outcomes, and it did. Now, we’re seeing who can benefit from relaxed cost-sharing where it optimizes health outcomes – for example, with members who have chronic diseases.


Lesson #5: Analytics is a constant journey

Our view is that we’ll never be advanced. The state of the art is evolving too rapidly. Analytics is a continual race. The data and analytics platform and our infrastructure are the largest capital investments that we’re making as a company. We completely restructured the organization around it. Since it’s the second time I’ve built a culture of analytics, we’re moving three times faster than the last time.


Lesson #6: Traditional reimbursement relationships may not matter as much as you think

To look at impacting payer and physician relationships, we partnered with a large health system to create a laboratory to test reimbursement arrangements. The interesting thing we found is that the reimbursement relationship between the payer / provider entity may not be what matters. It’s how the provider entity pays their individual clinicians that matters the most. So, now we’re looking at how we engage differently and downstream savings to the individual clinicians.

The other interesting thing is that the provider entities kept asking us for claims data. Thanks to our predictive models, we changed what data and tools we’re both using. Now, we’re working to get information into the actual workflow to reach the clinician directly.


Lesson #7: (Almost) everyone is in analytics

From a governance stand point, we don’t focus on individual initiatives. We focus on shared roles and responsibilities around data collection. A simple way to look at it is this: If your job is sourcing data, you’re moving into analytics. If your job is programming to allow us to access data (unless you’re the actuary), your job is moving into analytics. If your job is to maintain a repository of data, then your job is moving into analytics. There is no place to go in the organization now unless you’re tied to analytics.


Lesson #8: Make it impossible for the status quo to continue

Our lesson learned early on was that you cannot get an organization to change unless you eliminate the ability for the status quo to continue. Our first step was to eliminate the status quo. No one can go to the IT organization and say, “I want a database.” They can’t go to a vendor and ask for a new data source. It has to come through the Chief Analytics Officer so that we can support business needs and turn off the process of every business area creating their own arrangements.


Lesson #9: Simple changes can impact healthcare the most

In practice, change can be simple. For example, let’s look at open enrollment. Why not change open enrollment to ask the right three questions that end up being a better predictor of who’s going to have problems than any claims data?

If you’re going to issue an ID card, why not ask: Do you live alone? Do you have someone who goes to doctor appointments with you? Can you get to the doctor and pharmacy when you need to? Those three questions will end up predicting who is going to have problems far more than any claims data. But we don’t design our open enrollment process as a data-collection instrument. Well, we are now. And it’s not that much harder to ask three more questions. It’s easiest to test and scale this kind of change. We need the feedback to get the results we want.


Lesson #10: How do you accelerate the pace of change?

We saw that everyone wanted to change. The company, the providers, and the brokers all wanted to change. Maybe not the same change, but change was a goal. If the change is coming from the top down, it’s easier to get the company to buy in. There was some resistance to how we implemented change. But we created early wins by testing people’s business hunches to see what worked. It helped build trust. People could see that the results were better.


Steven Udvarhelyi, CEO, Blue Cross & Blue Shield Louisiana

Dr. Steven Udvarhelyi joined Blue Cross and Blue Shield of Louisiana as president and chief executive officer in 2016. He is a board-certified internist and has more than 25 years of experience in the health insurance industry.

Prior to joining Blue Cross and Blue Shield of Louisiana, Dr. Udvarhelyi was with Independence Blue Cross (IBC) in Philadelphia for almost 20 years, most recently serving as executive vice president, health services and chief strategy officer. Before IBC, he worked for Prudential Health Care in a variety of roles, including vice president, operations for Florida and vice president, medical services.

Dr. Udvarhelyi received an A.B. degree from Harvard College, an M.D. degree from the Johns Hopkins University School of Medicine and a Master of Science degree in Health Services Administration from the Harvard School of Public Health. Prior to his career in the managed care and health insurance industry, Dr. Udvarhelyi was a faculty member at Harvard Medical School with a focus on health services research.

Dr. Udvarhelyi currently serves on the National Board of Trustees for the Devereux Foundation. He has also served on the Board of Directors of NaviNet, which he chaired, the Board of Trustees of the Franklin Institute in Philadelphia, the Board of Managers for Tandigm Health,  the Board of Directors of NCQA, the Board of America’s Health Insurance Plans, the Institute of Medicine (IOM) Roundtable on Evidence-based Medicine and the IOM Committee on Comparative Effectiveness Research Priorities.

healthcare measurement and optimization

5 Executive Insights About the Future From 2018 Healthcare Events

What executives were talking about at the Health Evolution Summit and the Urgent Care Convention

Top healthplan executives see analytics in everyone’s future according to a roundtable discussion held during the 2018 Health Evolution Summit to trade insights about the future. Healthcare executives convened at a NextHealth Technologies event at the Summit to discuss the power of measurement and optimization as a key path to achieving sustainable cost savings.

Top takeaways offer insights into why a culture of measurement and optimization is becoming a critical core competency in today’s highly competitive environment. Healthplan executives are asking what it takes to build a culture of measurement that ultimately drives cost savings at scale.

Soon after the Summit, we also had the privilege of speaking to an engaged audience at the 2018 Urgent Care Convention. Urgent care professionals sought to know what works, make it better and demonstrate value faster for improved partnerships with healthplans and providers.

Here’s the roundup of advice that emerged from our conversations in both of these great venues.

Stop swinging for the fences and start getting on base

Too often the priority of the day drives conversation and attention, distracting executives from the hard work of incremental improvement. Innovation in healthcare means improvement first and foremost. There is tremendous discussion in the market around Amazon, CVS and Walmart. Yet these headlines are the exception rather than the rule. The more healthplans can focus on specific, incremental improvements the more they can deliver positive ROI. “We need to do a better job of delivering the right care at the right place at a lower cost,” noted Eric Grossman, CEO of NextHealth. Home runs are few and far between, usually higher risk and have a higher likelihood of attracting regulatory scrutiny.

We need to do a better job of delivering the right care at the right place at a lower cost.

Measurement is the backbone of competitive advantage

To get the small wins that add up to big wins, it takes a clear understanding of which programs are winners. One executive speaker noted that what gets measured gets done. Measurement is important because large employers demand it, and it’s a critical driver of sustainable competitive advantage. Measurement is how you drive impact in an organization.

Download our latest Insights Brief on what healthplan executives say is a top priority for 2018

Improvement funds innovation

Innovation is not as important as improvement in terms of impact on the business. It is the work of continuous process improvement that funds innovation, not the other way around.

Sustainable cost savings starts at the top

To drive change, you need to start at the top. It’s essential that leadership teams establish a culture where analytics isn’t an afterthought. It’s how you prove the business case, how you know what works to reduce medical costs and how you can tell if you’re serving members with the best programs.

Prediction is not the same as measurement and optimization

Being able to predict that an outcome is likely to happen to a certain population is powerful. Yet, it’s not the end of the story. Measurement tells you what worked so you can optimize and scale around the specific outcomes that actually worked.

Overall, we believe that healthplans are poised to go beyond basic insights. By analyzing more data more scientifically and using advanced technology to do it, we can know what works sooner to reduce medical costs. This speeds the process for getting members into high-impact clinical programs faster. It’s time to look at why so many healthplans are adopting advanced measurement and optimization practices to get to this next level of results. Best practices from your peers are included in our latest research brief. Check it out.

The Fast Eat The Slow

An Interview with Kristy Cunningham, Senior Vice President of Strategy and Insights, McDonald’s USA

How does an established market leader facing digital disruption and fierce competition take smart risks, place big bets and move fast to deliver top and bottom line growth?

McDonald’s USA Senior Vice President of Strategy and Insights, Kristy Cunningham, shared her insights from enabling McDonald’s strategic approach with Eric Grossman, CEO of NextHealth Technologies in the latest Analytics & Outcomes podcast. Listen to the full podcast.

Excerpts from the interview with Kristy include:

Listen and move quickly

Consumer perceptions can spread very quickly, especially with technology. To shape those perceptions and show that you’re innovative, you need to demonstrate that you’re listening to the customer and meeting those needs better than others. That means you need to move faster.

Put simply, the fast eat the slow.

The power of analytics

Any strategy that’s worth its salt is built on a set of facts that drive great insights.

Analytics is a key enabler of everything that we do. It has proven effective in taking the emotion out of it, which is important. Analytics also helps us balance the risk and reward. It equips us to decide if the size of the prize is big enough to take on a little extra risk and move a little faster.

And, it helps build organizational confidence. The analytics and economics that have come out of test markets have been really important in showing where change had high impact. Testing gave us more confidence so that we could move faster and take on more risks.

It takes discipline to make better decisions

To make the most of analytics, you have to be disciplined. For example, we loved the idea of the breakfast Happy Meal, so we put it in the market. Yet, it didn’t drive incremental sales. It drove trade-off purchases instead. We really wanted this offer to work, but we let the numbers and the analytics lead us to the right conclusion. In this case, it was not the right time, not the right offering.

Test and learn empowers bold actions that deliver results

Competitors try to chip away at all levels, all of the time. So, it’s important to listen to customers and move faster.

There are different ways to use a culture of test and learn to make risk taking more acceptable. We apply this all of the time for our mobile order and pay offers. You can try this vs. that, and see which has a better result. Small adjustments can add up over time.

Where McDonald’s has seen quite a difference and therefore better results is being able to take bold moves. That’s the real risk-taking. It’s not just the incremental steps, rather it’s how you place some pretty big bets. To place big bets, you need a good, disciplined test and learn approach and culture that can help people get over what may have been more conservative approaches in the past.

It’s Time To Know What Works To Control Healthcare Costs

It’s Time To Know What Works To Control Healthcare Costs

Selected insights from senior healthcare executive summit outlining how to use measurement and optimization to drive down medical and administrative costs.

Healthcare Executives: It’s time to know what works to control healthcare costs.

With healthcare costs rising at unsustainable rates, enterprise-wide cost control is a strategic imperative. At the same time, executives are investing 7, 8 and sometimes 9 figure budgets in clinical and consumer engagement programs without knowing specifics about which of these programs actually work to reduce costs.

Executives from 10 leading health insurers representing over 25% of the insured U.S. population gathered in March 2018 at Microsoft’s corporate headquarters to share common challenges and discuss pathways for progress toward more advanced measurement and optimization practices. A key goal of the meeting was to validate that these pathways would indeed provide a sustained competitive advantage.

Selected insights from the senior executives, who serve on the NextHealth Executive Advisory Council, are listed below.


Executive participants confirmed the strategic importance of building a culture of measurement. Driving optimization or continuous process improvement (CPI) requires executive commitment and sponsorship, according to these leaders. They further affirmed that leveraging a platform to test and learn what works helps drive scale while building a consistent language and measurement methodology to support CPI.

Executives agree: progress toward advanced measurement capabilities in healthcare is critical to stay relevant and compete effectively.

Executives agreed that progressing toward advanced measurement capabilities is critical to stay relevant and compete effectively. There is an inherent need to know what’s working faster and more confidently. At the same time there is often cultural resistance to progressing along this path. “My program is different and can’t be measured this way…” is a common sentiment.

Selected takeaways

  1. Measure and optimization is a top priority and garners budgets–organizations are investing in this.
  2. No organization believed they’ve achieved an ‘advanced’ level along the NextHealth Measurement and Optimization Value Curve.™
  3. Building a “test and learn” culture is essential to progress toward advanced measurement and optimization.

JP Morgan conference san francisco

Insights from the 36th JP Morgan Healthcare Conference

Healthcare executives need to know what works to make better, faster decisions.

Companies such as Amazon, Boeing, Microsoft, Netflix and Capital One run tens of thousands of experiments each year to pinpoint which products and programs work. They have to because they can’t afford to waste time and money on investments that don’t deliver revenue or cost savings. Collectively they’ve uncovered hundreds of millions of dollars of value by making decisions faster about what works and what’s a waste of time. And with Amazon’s recent entry into the healthcare space, health plans have no choice but to build effective test and learn capabilities and strong cultures of experimentation. Data-driven disruption just came from something looming on the horizon to knock on the front door.

Can you confidently and quickly isolate which of your programs are driving impact so you can optimize them ?

That was one of the key questions on the minds of healthcare executives at the 36th Annual JP Morgan Healthcare Conference that took place January 8-11, 2018 in San Francisco, CA.

NextHealth Technologies attended the conference and co-hosted a dinner with Deloitte and Norwest Venture Partners for national and regional plan executives to discuss affordability and how plans can tackle the unsustainable cost environment.

Two key themes emerged from the conference and dinner event:

1. Urgent need to know what works

Attendees at the dinner agreed there is an urgent need to better know what is working when it comes to reducing costs and engaging members. Yet plan executives and investors polled at the conference felt that health plans in general are struggling to prove ROI of all their investments. Which ones are worth the effort and which are a waste of time?

Plans are trying desperately to halt the tsunami of rising costs and in the rush to try many things at once, may actually be driving up costs by investing in programs that don’t work. The net result is an environment that fails to generate the critical insights about which investments work, which investments are a waste of precious resources and why.

By running tests to prove what works and what doesn’t you’ll widen your band of confidence to make better decisions, faster.

2. Differing levels of analytics maturity

When it comes to data and analytics – a key focus for healthcare executives in 2018 – there were widely diverse levels of maturity.  Some attendees and plan executives are working on data cleanliness and integrity while others have already invested “into the nine figures” in toolsets and programs aimed at tackling the affordability challenge.

Even if they don’t have perfect data today, plans agree there’s a pressing need to start using the data they currently have to move towards a test and learn culture and focus on knowing what works. The stakes are too high to not know what works.

Start today to make ROI and measurement part of your culture while you continue to take concrete steps toward cleaner data and personalized analytics.

NextHealth’s Perspective

  • Unlock transformative value by making better decisions faster

The current environment of trying everything in hopes that something works is unsustainable. The data and analytics horsepower currently exists so that healthplans can affordably prove which interventions and programs drive down costs within a matter of weeks. Solutions such as NextHealth easily augment and amplify current analytics programs to isolate and increase what works faster.

  • Create a culture of analytics

Adopt a posture of measure, know and optimize. Plan executives need to measure their efforts to know what works and then optimize their investments to do more of what they know works. It all starts by creating a culture of experimentation and analytics. It doesn’t matter where you are on the maturity curve – what matters is that you start testing and start learning. “By running tests to prove what works and what doesn’t,” said NextHealth CEO Eric Grossman, “you’ll widen your band of confidence to make better decisions, faster.”

It doesn’t matter where you are on the maturity curve – what matters is that you start testing and start learning.

  • Start wherever you are

Attendees at the conference and the roundtable dinner represented multiple points along the spectrum of data and analytics maturity. Some had already invested millions of dollars in programs and tests around consumer engagement. Some were still focused on cleaning up their data warehouses to ensure they had good data for personalization. Start where you are and move toward formalizing tests, learn from the results to know what works, do more of what works and scale those programs for cost saving across the entire business.

  • Optimize for what you know works

The crux of the opportunity is less about data quality debates and more about discussions of how to build a culture that is willing to make decisions and optimize based on what works. A true measure of a culture of innovation is the ability to drive value from what works and being unafraid to turn off what doesn’t and try again.

Once you have good data to measure what works and know what is effective then it’s time to optimize for what works. By now many people are familiar with the Facebook/ Snapchat rivalry where Facebook copied the popular Stories feature of Snapchat soon after parent company Snap went public. The net result was and continues to be devastating competitive pressure for Snap.

What many people don’t realize is that Facebook began it’s assault on Snap in France with a small test. The company rolled out a test of the Stories feature, proved that it worked and then started to scale to the larger Facebook platform. But even then it wasn’t a sure thing. It took multiple (failed) iterations to figure out the right combination of features and placement before the feature won widespread adoption.

The company didn’t stop running tests and trying new combinations until it found the one that worked and could prove it with test data.

NextHealth provides the scientific rigor and analytical insight to help plan executives know what works to reduce costs at scale.

See how the NextHealth analytics solution reduced avoidable emergency room visits by 25%.

Finding the Signal in Difficult Data

Customizing Algorithms to Drive Better Outcomes

By Dr. Douglas Popken,
SVP of Analytics


Changing consumer behavior is complicated. Many different factors can impact human health-related behavior, both systematically (weather, changing regulations, general economic conditions, cultural influences, etc), and through individually based random variations (health status, mood, prior-held beliefs, financial circumstances, etc). Healthcare marketing teams craft messages targeted at consumers to try to influence consumer behavior, but even when they see impacts, they often cannot pinpoint whether it was their messages or perhaps another factor that caused the change. It then becomes difficult, if not impossible, to know how to efficiently allocate intervention resources.

At NextHealth, we designed our automated platform and methodology to address this critical issue. NextLift, a major component of the platform, includes a module that calculates both the lift and the statistical significance of each campaign or “nudge” impact on consumer behavior.  By using randomized controlled trials (RCT’s) we can isolate whether the change is due to the nudge itself or some other systematic factor.  By considering significance, we can also judge whether the measured lift is simply a reflection of random variations in the data.

In statistics, one way to distinguish an underlying effect from random variation is to use a measure called “p-value”.  P-value is a probability – the probability of observing a lift value more extreme than what was observed if the “null hypothesis” (that the true lift is equal to zero) were true.  P-value provides a scientific basis for claiming causality.  To claim success, the p-value needs to be small; at NextHealth, we use as a baseline statistical standard that if p <= .05, then the observed lift can be attributed to the nudge.

Lift is measured via a statistical “comparison of means” test, where the mean KPI (Key Performance Indicator) for the trial group is compared to that of the control group.  The standard statistical approach for determining the significance (and confidence intervals) of a comparison of means test is some form of “two-sample t-test”.  The drawback to unmodified use of these traditional techniques is that they require assumptions about the nature of the underlying data, especially, that the data follows a normal distribution.  However, the medical utilization and cost data typically encountered with our clients is often highly skewed with both a long right-hand tail and many zero values.  On the other hand, the two-sample t-test is known to be highly robust to non-normality if the data sets are large enough and/or the data is not too severely non-normal.  In practice, it is difficult to know when these conditions have been met.  For these reasons, NHT now uses a modern, robust methodology known as the bootstrap technique (Efron and Tibshirani, 1993) for determining the p-value and confidence intervals for the lift value.

The bootstrap method is based on repeated resampling to simulate comparison outcomes, with samples drawn from an empirical distribution of the observed trial and control data.  Its key advantage is that it requires no assumptions about normality and is therefore highly appropriated for skewed medical data.  It reduces the motivation to transform the data before analysis to make it less non-normal (e.g log transformations or trimmed mean approaches), allowing for direct comparison of the true means of the two groups.  To compute p, we use bootstrap samples of the t-statistic (each of which is computed as described below) for comparison of the population means of the two groups.  A variance stabilization technique is automatically applied to achieve the highest accuracy.  To compute confidence intervals on the lift, we use a bootstrap distribution of mean lift values with bias and skewness corrections known as BCa.  The specific techniques we use are described in greater detail in Barber and Thompson (2000).  See also Efron (1987).

To compute the t-statistic for each bootstrap sample, NextHealth uses a modification of a well-known statistical test for comparing the means of two populations with unequal variances, “Welch’s t-test” (see  or  Other methodologies are available, but research has shown that Welch’s t-test generally provides the most accurate results for the type of data we typically work with (Fagerland and Sandvik, 2009).  The modification NextHealth has made to the standard Welch’s t-test is to weight the observations in each group by the duration of the observation period (in years).  One immediate advantage is that the duration weighted mean is equivalent to the population mean KPI per member year (total KPI value/total member years), which is the most relevant statistic for our analyses.  Another motivation for weighting is that our underlying observations are annualized, but the observations themselves have variable observation periods.  Without weighting, a non-zero observation resulting from a short observation period can become relatively large when annualized, thereby causing the dataset to have unusually high variances, particularly in the early stages of a campaign/program.  A similar weighting approach is described in Bland and Kerry (2008) for a t-test that assumed a single pooled variance for the samples.  To compute the weighted means and weighted variance of each group, we rely on equations described at, in the subsection, “Weighted Sample Variance…Reliability Weights”.   The weighted means and weighted variances computed from these equations then replace the unweighted mean and variance parameters described within the Welch’s t-test.

Measuring program success in the face of skewed outcome data with a high degree of random variation is difficult.  NextHealth has employed a combination of the best statistical methodologies available to achieve the highest degree of accuracy.


  • Barber, J.A. and Thompson, S.G.  Analysis of cost data in randomized trials: an application of the non-parametric bootstrap.  Statistics in Medicine, 19, 3219-3236.
  • Bland, J.M. and S. Kerry.  2008.  Weighted comparison of means.  BMJ, 316, 129.
  • Efron, B.  1987.  Better bootstrap confidence intervals (with comments).  Journal of the American Statistical Association, 82(397), 171-200.
  • Efron, B. and Tibshirani, R.J.  1993.  An Introduction to the Bootstrap.  Chapman and Hall, New York
  • Fagerland, M.W. and L. Sandvik.  2009.  Performance of five two-sample location tests for skewed distributions with unequal variance.  Contemporary Clinical Trials, 30, 490-496.

Elise Mariner

Elise Mariner, Regional VP, Sales & Business Development - West

Elise Mariner is the Vice President of Sales and Business Development in the Western region for NextHealth Technologies. Previously, she was the VP of Client Services, responsible for all client implementations and measurable results by collaborating closely with her client stakeholders in care management, marketing, population health, and analytics. She has driven significant outcomes for clients in use cases including Avoidable ER and Out-of-Network reductions as well as gap in care closures.

Elise has spent her career in healthcare and is passionate about empowering every individual to access the right care, at the right place, at the right time. She has a background in engineering, business and public health and is able to combine skills across these disciplines to help her clients achieve outcomes using the NextHealth Technologies platform. Early in her career she designed next-generation medical devices to treat heart disease. Prior to joining NextHealth, Elise was a healthcare consultant guiding provider organizations through important strategic and operational decisions and also experimented in the direct-to-employer healthcare delivery space.

Elise and her husband enjoy active pursuits such as skiing, snowshoeing, biking, and renovating houses.

B.S. in Biomedical Engineering from Columbia University, and MBA/MPH from UC Berkeley


Test and Learn Your Way to Medical Cost Savings

To be truly innovative you need to create a culture of experimentation.

Have you ever bought anything from Amazon zShops? Probably not.

How about Amazon Auctions? Not likely.

You are not alone because these two massive experiments were quickly scuttled by the retailing giant 15 years ago on the road to hitting their home run with Amazon Marketplace where nearly 50% of current units sold on are from third-party sellers. zShops and Auctions were not failures; rather, the company had to test these two concepts in order to learn how to successfully execute the Marketplace offering. A test and learn culture is critical for innovation not just in retail but in every industry, including healthcare.  

“To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment,” says Amazon’s CEO Jeff Bezos in his 2015 shareholder letter. “Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”

“Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”  – Amazon CEO Jeff Bezos

NextHealth CEO Eric Grossman recently wrote about why it’s important that healthcare leaders transform their organizations into test and learn cultures in his recap of the 2017 Oliver Wyman Health Innovation Summit. There are important lessons in experimentation and cost savings for healthcare industry executives from other industries including retail and financial services.

In a recent discussion during the “Finance Disrupted” event by The Economist, Capital One co-founder and QED Investors Managing Partner Nigel Morris detailed how the competitive advantage for companies today lies not just in having lots of data to run experiments but rather how companies leverage that data for insights by building a test and learn culture. Morris explained it best in his outline of Capital One’s strategy:

“The core idea was that the credit card business is not really the traditional lending business, it’s not really banking at all. What it really is is the leveraging of information in order to be able to put the right product in the right customer’s hands at the right time. The way you did that was in two or three ways. One was amassing huge amounts of data at the customer level and then using experimental design and testing different product and market combinations in order to optimize value to the consumer and net present value to the entity.”

Sound familiar?

You could replace “credit card business” in the above quote with “healthcare industry” and be accurate in describing the opportunity for health plans to build test and learn cultures that better understand and align the right products for the right members at the right time. Capital One now runs over 80,000 tests per year to pinpoint the ideal combination of product, customer, and timing.

For Amazon, Capital One, and a growing number of health plans, when it comes to reducing costs, growing revenues, and improving outcomes, the insights from the tests and how (and how quickly) they are applied are what matter most. Through thousands of experiments, Amazon learned that customers are already using its artificial intelligence framework for early disease detection that saves lives and lowers costs. Through testing and learning, CapitalOne found that people who complete an application in all capital letters pose a higher credit risk (i.e. higher costs). 

Here is how Jeff Bezos described a test and learn approach for Prime Now that was put in place in a matter of months:

“Prime Now offers members one-hour delivery on an important subset of selection, and was launched only 111 days after it was dreamed up. In that time, a small team built a customer-facing app, secured a location for an urban warehouse, determined which 25,000 items to sell, got those items stocked, recruited and onboarded new staff, tested, iterated, designed new software for internal use – both a warehouse management system and a driver-facing app – and launched in time for the holidays. Today, just 15 months after that first city launch, Prime Now is serving members in more than 30 cities around the world.”

How does NextHealth help its health plan customers test, learn and optimize outcomes quickly?

NextHealth’s Measurement and Optimization platform automates health plans’ ability to quickly test, measure and optimize any clinical program. Key benefits include:

  • Faster, data-backed insights allow teams to get to better business decisions quickly and with greater consensus

  • Standardized and consistent analytics methodologies and program set-up help elevate the organization to a shared conversation based on experimentation, building a culture of measurement

  • Automating the ability to test, learn and optimize allows talented analytics and healthcare economics resources to do more with less and operate at their highest capacity

The platform has enabled clients to measure existing programs in as little as 60 days, learning which programs are effective and which are not, gleaning administrative cost savings quickly. New programs, such as reducing avoidable ER utilization, EPDST, HEDIS improvements, and others, can be added to the platform as well, delivering medical cost savings using machine learning to optimize outcomes by assigning members to the programs that are most likely to work for them. NextHealth’s managed services offering, included in every engagement, ensures that clients have access to the learnings gleaned from our experience, furthering speed to value.

Is your organization reducing medical and administrative costs through a test and learn culture?

Our platform accelerates data-driven decisions and enables a culture of measurement. See how.