Data-based impact evaluations are an essential way for nonprofits to understand how their work is affecting the people they aim to help. But how do we determine what constitutes an improvement? What is our yardstick for success? In many cases, impact evaluations look at improvements in income, health, or life expectancy. But what about equity in impact evaluations?
Right now, we’re working with a few different projects that are trying to increase household incomes and financial inclusion. These are both noble goals. And these projects are run by good people, doing good work.
Unfortunately, equity in impact evaluation is often overlooked. But there are several minor changes we can make to these impact evaluations to shift the focus from simple income increases to increases in equity.
Shifting the Focus
For one of the projects we’re working on, we have access to a dataset that shows changes in household income over time in the supportive project. And if we do a traditional impact evaluation, the results look like this:
- Income improves for both the treatment and control group
- Income increases more for the treatment group than the control group
It appears the project is a success.
|Control Group||Treatment Group|
But what if we look more closely at the social infrastructure in which the participants live? The story changes when viewed through the lens of equity.
For example, the people included in this project identify as members of three distinct groups. And these groups do not have the same experience. They started at different income levels, and their incomes increased at different rates throughout the project.
|Control Group||Treatment Group|
|Ethnic Group 1||$90||$90|
|Ethnic Group 2||$105||$101|
|Ethnic Group 3||$112||$110|
|Ethnic Group 1||$97||$95|
|Ethnic Group 2||$114||$126|
|Ethnic Group 3||$121||$159|
Within the treatment group, the average income increase was $27. But when we drill down, there is more to this story. Average income for the first ethnic group increased by $5. The second group saw an average increase of $25, and the third, $49. That’s a 45% increase for the group that started in the most privileged position. And only a 6% increase for the poorest group.
In fact, when we look at changes in the income gap rather than simple income increases, it appears our project widened the gap. And that’s why considering equity in impact evaluations is so important.
|Distance from the richest Ethnic Group|
|Ethnic Group 1||$22||$20|
|Ethnic Group 2||$7||$9|
|Ethnic Group 1||$24||$64|
|Ethnic Group 2||$7||$33|
Equity in Impact Evaluations Matters
Considering equity in impact evaluations isn’t just about money. For this example, I used income because it’s easy to illustrate the problems. But we need to apply our equity lens to all data product results. And not just ones that appear successful. Changes that look positive can actually be increasing inequality and vice versa.
Sometimes an initiative that doesn’t appear to have made a great impact at first glance actually looks better when we consider equity improvements.
Let’s say you’re using a social media campaign to increase hospital visits. There are services available at a local hospital that could help the local population, but it doesn’t seem like people are using them. You need to examine how the number of visits changes and if who is visiting changes.
Maybe the number of hospital visits doesn’t increase significantly. Were your efforts in vain? If there was a big gap in education levels in the community and you managed to reach a lot of people who don’t speak the primary local language, you may have increased the equity. (More people who were historically disadvantaged are now using the services.)
If you don’t carefully define your key outcomes, there’s a very real chance you could miss what you’re looking for. Make sure what you measure if what you care about. Don’t make the mistake of thinking an improvement in one metric is a success — or that a lack of change in that metric is a failure.