Do Lower Readmissions at Penalized Hospitals Signify an Effective Policy or Simply a Statistical Anomaly?

In 2010 Medicare introduced the Hospital Readmission Reduction Program (HRRP), one of a series of reforms under the Affordable Care Act aimed at improving quality and lowering costs across the healthcare sector. The program imposes penalties on hospitals if they have higher than average 30-day readmission rates for certain conditions.

Since HRRP was enacted, risk-adjusted readmissions for the conditions targeted have declined, especially among hospitals that were penalized. While some interpreted this trend as evidence the policy was improving outcomes, USC Schaeffer Center Expert and Price School of Public Policy Professor Neeraj Sood wondered if it was simply an example of a statistical phenomenon known as regression to the mean—in an outcome measure with an element of chance, the likelihood of a repeat extreme outcome is low when measured multiple times.

A new analysis led by Sood and published in JAMA Internal Medicine shows that at least three quarters of the improvement in readmission rates by hospitals who had poor baseline performance was due to regression to the mean (i.e. statistical luck) rather than the policy.

What is Regression to the Mean

Regression to the mean is a known statistical phenomenon. It occurs when an outcome is measured multiple times. Outcomes that are extreme relative to the statistical average, or mean, during the first measurement are more likely to be closer to the mean in subsequent measurement periods simply by chance, because more extreme values have a lower probability of occurring.

What is important when considering whether regression to the mean might play a role is whether the variable being measured has a component that can be considered random, or chance.

“We knew that some of the improvements in readmissions at penalized hospitals were due to regression to the mean as readmissions are determined both by care quality and random events such as a patient falls or bad weather. We wanted to know how much of the improved readmissions were due to regression to the mean versus improvements in quality of care. Answering this question is critical for understanding the true effects of Medicare’s decision to penalize hospitals for excess readmissions.”

There are many competing factors when considering why a patient might experience a readmission following a hospital stay. Random events, such as random biological variation in treatment efficacy or severe weather or a patient fall, might play an important role in addition to factors like quality of care and post-discharge treatment plans.

Penalties for Hospitals with Below-Average Baseline Performance Are Determined in Part by Chance

Sood and his colleagues used 2006 through 2014 Medicare data to identify hospitalizations and 30-day readmission episodes. Over 3000 hospitals were included in the study.

Medicare started penalizing hospitals with excess readmissions for heart failure, acute myocardial infarction, and pneumonia in fiscal year 2013. To identify hospitals considered to have excess readmissions, they used fiscal years 2008 through 2011 as the baseline.

After calculating the change in readmission rates (based on the baseline 3 years of 2008-11) for hospitals that were deemed to have excess readmissions in 2013, Sood and his colleagues ran a series of thought experiments to tease out whether the policy incentives really had an effect on the outlier hospitals that reported excess readmissions and were consequently penalized.

For example, they set up a scenario to measure the effect of an artificial HRRP treatment 3 years before the actual program was implemented and observed changes in excess readmissions as if the policy were in effect.

They found that hospitals with below-mean performance during the 2008-2011 baseline window experienced a decline of 4.8 to 7.1 percentage points in excess readmissions during the subsequent 3 years. However, 74.3 percent to 86.5 percent of the improvement observed was explained by regression to the mean. Similarly, hospitals that had better than average baseline performance during the baseline period experienced an increase in the subsequent reporting period, of which 83.6 percent to 91.8 percent could be explained by this statistical anomaly.

“The evidence supporting this conclusion includes the fact that we observed similar changes in excess readmissions at below-mean and above-mean performing hospitals when we defined performance during an alternative measurement period predating the HRRP and when we examined changes in performance going backward rather than forward in time,” write the authors.

These findings mean that hospital performance and the penalties associated with the HRRP are strongly influenced by chance. Given this, modifications to the program that better account for the role of chance might improve long-term effectiveness of the program.

“The findings suggest that some hospitals are being penalized not because they provide poor quality care but because they were simply unlucky. These hospitals might find it difficult to sustain programs to improve quality,” explained Sood.

In addition to Sood, this study was authored by Sushant Joshi, Teryl Nuckols, Jose Escarce, Peter Huckfeldt and Ioana Popescue. Funding for this study was provided by AHRQ.