What is the role of quantitative surveys in shedding light on different lived realities?
Which choices made during the design of an impact evaluation affect who is represented and listened to?
Data collections can make structural inequalities invisible and these are all questions which we regularly ask ourselves whilst working as advisors in impact evaluations for Oxfam GB. We do impact evaluations as a learning and accountability mechanism to enhance programme quality. In doing so, we acknowledge that power matters in evaluation – and that the choices we make as evaluators are not neutral. And yes, for us too equity in impact evaluations matters!
Gender is one dimension of power which also intersects with other dimensions such as class, race, sexuality, ethnicity, etc. Gender shapes relationships, access to resources and decision-making, including at a personal level, within the household, and more broadly. Feminists – and feminist economists in particular – have said it for a while, and still, we have observed a tendency in data collection processes to focus at the household level as a unit of analysis without considering intra-household dynamics. Sometimes, survey protocols rely on hearing from respondents already in positions of power in the household, hence reinforcing patriarchal norms. From a statistical perspective, we have to be intentional in our sampling strategies to enable representation and visibility of different social groups, and make statistical analyses by social group possible.
As part of our Effectiveness Reviews, we have embarked on a journey of integrating a gender lens at the core of our impact evaluations. Which means – among other things – hearing from women and men! By doing so, we make sure to systematically look at gender differences and test for differential impacts of Oxfam’s programmes.
And in practice?
Well, we have tried two different sampling strategies. The first strategy is inspired by the Women’s Empowerment in Agriculture Index, and consists of surveying several household members, women and men, for individual surveys, within the household. As much as possible, the individual surveys take place at the same time to ensure privacy. Respondents are then brought together to complete the household survey. You can see an example of how this sampling approach was used in practice during an impact evaluation here, and we also shared a post on Oxfam’s REAL Geek series which you can read here in which we discussed the pros and cons of using this sampling strategy compared to the following one.
The second strategy consists of randomly varying whether to survey a woman or a man in each household. You can see an example of how this approach was applied in practice here. Before going into the details of the protocols we developed, it is critical to highlight here that we regularly work in contexts where a comprehensive roster of individual household members is not available, nor is it feasible to conduct the full listing prior to data collection. If you are carrying out evaluations where such rosters are available or can be collected, you will draw your sampling frame before carrying out the survey and you may want to skip to the last paragraph of this blog! If you want to hear more, please carry on reading.
As a first step, we define the main respondent irrespective of gender (for example adult household members involved in certain activities). As a second step, because we acknowledge that interviews are a social interaction and recognize the role of power dynamics – and gender one in particular – in such an interaction, two different protocols may be followed depending on the context in which the survey takes place and the content of the questionnaire.
The first protocol is enabled by technology and is irrespective of the gender of the enumerators. Using digital data collection, the survey software can randomly allocate whether the respondent of the survey should be a woman or a man, each time a survey form is open. In the second protocol, we want to match the gender of the interviewer and interviewee. One way of doing this is to randomly allocate the gender of the interviewer in charge of interviewing a given household, which will then determine the gender of the person to interview.
In both protocols, there are implications on the composition of the team of enumerators. We usually aim to have a team which is balanced equally in gender (half women and half men). There are also implications on the flexibility of the team during the survey. In a lot of cases, availability of respondents is not gender-neutral. This means that enumerators may have to come back at the time when the selected respondent is available to complete an interview. Also, in order to match the gender of the interviewer and interviewee the team may need to redeploy enumerators during data collection if the household was composed of no one who could be the main respondent of the randomly identified gender.
Is that it?
While sampling is critical to enable visibility and representation, the integration of a gender lens also means changing our measurement and analytical tools – we are currently working on guidelines so do stay tuned!
Of course, there are limitations, some of which have already been mentioned here, but let me add two more.
The focus on women and men in these analyses carries a risk of essentialization – fixing and naturalizing the meaning of social categories. Feminist quantitative social scientists have written about this (Sigle-Rushton, (2014)), and two co-workers and I touch on it here, while reflecting on feminist values in Monitoring, Evaluation and Learning practices at Oxfam GB. One way to overcome this would be to adopt intersectional analyses, to acknowledge that social groups are not homogeneous and that the intersection of power structures shape specific positions and experiences, in a given context.
The second limitation to mention is that by focusing on the binary, we are making different realities invisible. While the binary is convenient from a statistical point of view, statistics and survey protocols are the means, not the end. We are currently grappling with this issue and for now, I recommend this read from Edge Effect and 42 Degrees from the gender equality and resilience winter school.
We are working on these two points and hope to make more progress soon. Feel free to get in touch if you have any questions or comments on this topic!