A special dispatch from AcademyHealth’s Annual Research Meeting in Seattle, June 2023
This post recaps a panel discussion focused on Measuring Impact of Policy Strategies on Health Equity. I was fortunate to be among the panelists. My talk focused on 7 common mistakes when conducting evaluation and health equity-focused research. In this post, I share a brief summary of the points I made.
Mistake #1: Looking at “disparities” through a “special populations” lens
Who is “special”? Women, kids, older adults, racial/ethnic minorities, SOGI minorities, people with multiple chronic conditions, people living in poverty, veterans, people with stigmatizing conditions like obesity or MH/SUD, people with Limited English proficiency… basically everyone is special in some way, so…-
Instead of special populations, look at specific populations
-
-
Instead of disparities, we frame them as inequities; the opposite of health inequity is health equity
-
Health equity is not supported by equality of resources; specific populations need specific extra resources to have the same opportunities to attain good health
-
Mistake #2: Not fully accounting for area-level conditions
-
Place matters!
-
In a large multi-state Medicaid innovation evaluation, we showed that even in an all-Medicaid population, people living in different neighborhoods had different outcomes – in the highest-risk communities, Medicaid enrollees had more ED visits, more hospital stays, and fewer PCP visits.
-
We need to account for more than the usual limited set of predictors – eg, poverty rates can be similar across two areas that have very different access to care based on provider and hospital/clinic availability
-
Our Local Social Inequity (LSI) scores draw on 200+ measures and the latest AI methods to generate risk scores predicting specific health outcomes – life expectancy, drug overdose mortality, cancer mortality, sexual and reproductive health, and more – we have a website (RTIRarity.io) where you can see our scores across the US
-
Our scores explain more variance than any other index we’ve analyzed
-
More health outcome variance explained = more baseline SDoH risk accounted for, making it easier to see individual variance
-
Can also use as a stratifier, evaluate equitable access to care, help guide resource allocation (eg mobile family-planning clinics)
-
Mistake #3: Reference groups
-
Example 1: The default is nearly always White, but consider making it the group with the best outcomes! Or, compare each group to all others, so you can get relative ORs for everyone.
-
Example 2: The default is nearly always male, but what if they are doing worse? Hard to interpret negative ORs, so choosing wisely helps with interpretable findings.
Mistake #4: Assuming the data you want will be available, accurate, and complete
-
Expect the unexpected – eg, 80% missing race/ethnicity in Massachusetts MassHealth data
-
Collect better data where possible
-
Fill in with other data (for example: RTI Rarity links more than 40 public-sector databases across Federal government, NGOs, academia)
Mistake #5: Imputing race/ethnicity
-
Is there a valid model to predict race/ethnicity? How could there be, given it is such a complex social construct? Do these models really pass the face validity test, especially among affected populations?
-
Assumptions in the surname-based methods are counter to equity, given cross-racial/ethnic adoptions and marriages
-
Consequences of “guessing wrong” are potentially making policies that reinforce the status quo and decrease health equity
Mistake #6: Assuming members of racial/ethnic communities are all poor/deprived/vulnerable, have worse outcomes, mistrust the healthcare system, won’t participate in clinical trials
-
Don’t assume – ask!
-
There can be health benefits of living in a racial/ethnic enclave; the real problem is often a lack of resources
-
Check your language and mindset for stigmatizing, othering terminology
Mistake #7: Not accounting for non-parallel trends
-
Difference-in-differences method requires parallel trends – we assume that the change measured in the comparison group between the pre-intervention and post-intervention periods approximates what would have occurred in the intervention group had the intervention not occurred (the “parallel trends assumption”)
-
51 CMMI models posted between September 2012 and May 2021, 40 used DiD, 23 of the evaluations using DiD tested for parallel trends, 15 (79%) found evidence against parallel trends for at least some DiD models
These certainly aren’t all of the possible mistakes, but if you avoid these 7, you’ll be farther along than most!
Lisa M. Lines
Senior health services researcher at RTI International
Lisa M. Lines, PhD, MPH is a senior health services researcher at RTI International, an independent, non-profit research institute. She is also an Assistant Professor in Population and Quantitative Health Sciences at the University of Massachusetts Chan Medical School. Her research focuses on social drivers of health, quality of care, care experiences, and health outcomes, particularly among people with chronic or serious illnesses. She is co-editor of TheMedicalCareBlog.com and serves on the Medical Care Editorial Board. She served as chair of the APHA Medical Care Section's Health Equity Committee from 2014 to 2023.
Views expressed are the author's and do not necessarily reflect those of RTI or UMass Chan Medical School.