Better for Who? Examining Equity in Impact Evaluations
Data-based impact evaluations are an essential way for nonprofits to understand how their work is affecting the people they aim to help. But how do we determine what constitutes an improvement? What is our yardstick for success? In many cases, impact evaluations look at improvements in income, health, or life expectancy. But what about equity in impact evaluations?
Unfortunately, equity in impact evaluation is often overlooked . But there are several minor changes we can make to these impact evaluations to shift the focus from simple income increases to increases in equity.
Shifting the Focus
For one of the projects we’re working on, we have access to a data set that shows changes in household income over time in the supportive project. And if we do a traditional impact evaluation, the results look like this:
- Income improves for both the treatment and control group
- Income increases more for the treatment group than the control group
It appears the project is a success.
But what if we look more closely at the social infrastructure in which the participants live? The story changes when viewed through the lens of equity.
For example, the people included in this project identify as members of three distinct groups. And these groups do not have the same experience. They started at different income levels, and their incomes increased at different rates throughout the project.
Within the treatment group, the average income increase was $27. But when we drill down, there is more to this story. Average income for the first ethnic group increased by $5. The second group saw an average increase of $25, and the third, $49. That’s a 45% increase for the group that started in the most privileged position. And only a 6% increase for the poorest group.
In fact, when we look at changes in the income gap rather than simple income increases, it appears our project widened the gap. And that’s why considering equity in impact evaluations is so important.
Equity in Impact Evaluations Matters
Considering equity in impact evaluations isn’t just about money. For this example, I used income because it’s easy to illustrate the problems. But we need to apply our equity lens to all data product results. And not just ones that appear successful. Changes that look positive can actually be increasing inequality and vice versa.
Sometimes an initiative that doesn’t appear to have made a great impact at first glance actually looks better when we consider equity improvements.
Let’s say you’re using a social media campaign to increase hospital visits. There are services available at a local hospital that could help the local population, but it doesn’t seem like people are using them. You need to examine how the number of visits changes if who is visiting changes.
Maybe the number of hospital visits doesn’t increase significantly. Were your efforts in vain? If there was a big gap in education levels in the community and you managed to reach a lot of people who don’t speak the primary local language, you may have increased the equity. (More people who were historically disadvantaged are now using the services.)
If you don’t carefully define your key outcomes, there’s a very real chance you could miss what you’re looking for. Make sure what you measure if what you care about. Don’t make the mistake of thinking an improvement in one metric is a success — or that a lack of change in that metric is a failure.
Originally published at https://weallcount.com on March 1, 2019.
The We All Count project share examples, build tools, and provide training and education aimed at helping better understand data — so we can make it more transparent and fairer for everyone. Because when you do the math, we all count.
Sign up for our newsletter ‘The Lowdown’. Delivering plain language, no-jargon, in-depth articles about current issues in data science equity; video tutorials on how to find and correct these errors; interactive web tools that can help you bring your data game to the next level; and a deep dive into the ‘Data Life Cycle’, our keystone concept for thinking about the future of data.
