P-Value Won’t Set Us Free

By: Doug Popken, SVP Analytics

NextHealth has been very successful in developing the ability to measure attributable program lift across a variety of KPIs. One of the key challenges in doing so has been to create ways to accurately compute measures of uncertainty (confidence intervals, statistical power, and p-value), even when the KPI data is noisy, skewed, and full of zero-values. The modern, statistical bootstrap method and Bayesian approaches have been critical to accomplishing this, and they have been deployed in the platform. Clients have been very excited to see the statistically sound results, with one even going so far as to say, “P-value is the new ROI”!

But it is also true that statistically significant results are not always attainable, even when the “true” underlying impact of a program is positive. There may not always be enough members or enough measurement time to fully assess the KPI in question. Or there may not ultimately be any impact to find. For example, a “total cost of care” KPI can be highly variable and require thousands of member years (or more) to obtain a good measurement; however, it may not be possible or practical to gather that amount of data.

The campaign optimization component of the NextHealth platform already makes allocation decisions based on impact measurements, and it can do so without the need for statistical certainty.  It evaluates the distribution of results for each campaign (see figure below), and then allocates the next round of interventions to maximize expected impact according to the uncertainties.

Figure 1.  KPI Lift Distributions by Campaign

Now let’s look at the somewhat similar situation faced by a human decision-maker (DM).  Suppose that a program has been measured in the platform, and at the end of that program, we see some positive ROI, but it is not quite statistically significant. That is, the lower confidence limit (LCL) on the mean ROI is below zero.  See Figure 2 for an example.

Figure 2.  Example of Non-significant Positive ROI

Now, the decision-maker must decide whether to continue the program, and as a part of that decision, needs to know what ROI is reasonable to expect from it in the future. The decision will partially depend on the amount of risk that the DM wants to take on. In other words, what level of confidence is “sufficient”? 90%?  80%? Some other value? The DM also needs to know the minimum ROI associated with each level of confidence. It turns out, it is not difficult to transform a ROI distribution, as shown in Figure 2, to provide the needed data.  Without going into all the details, the result of the transformation is shown in Figure 3 (below), which is called an “ROI decision curve”.

Figure 3.  ROI Decision Curve

An ROI decision curve would allow the user of the platform to associate a minimum ROI (horizontal axis) with a probability (vertical axis). For example, there is a 92% chance the ROI is positive, but there is an 80% chance that the ROI is at least $55K. None of these statements require knowledge of p-value or whether the program even had statistically significant results. They are all a consequence of the probability distribution describing the lift, and consequently, the ROI. The user pegs their ROI assessment to the level of risk that they are comfortable with.

If you’d like to discuss ROI Decision Curves and how NextHealth’s analytical solution can help your organization optimize outcomes, please contact us.