Icon: Survey data collection
When a new nutrition intervention is introduced, stakeholders need to monitor the progress of scale up. However, there is often a lag between the introduction of an intervention and its integration into national data systems, including household surveys and administrative platforms. Valid coverage indicators are also needed by program evaluators, implementation scientists, and modelers who use intervention coverage as an outcome or a model input.
Investment in coverage indicator design, testing and uptake is not often prioritized as part of global and national efforts to roll out new interventions. A lack of clear recommendations for how to collect valid coverage data leads to missing or poor-quality information about who is being reached.
Population-based coverage of maternal and child nutrition interventions is measured in household surveys by asking survey respondents about the interventions they or their children have received. It can be challenging to design a concise set of questions that respondents can understand and respond to accurately and that are feasible to implement in the context of a larger survey. A previous DataDENT blog highlighted challenges specific to measuring iron folic acid (IFA) coverage; similar challenges apply to measuring coverage of other nutrition interventions.
DataDENT has contributed to the design of new coverage indicators for maternal, infant, and young child nutrition (MIYCN) counseling, micronutrient supplementation during pregnancy, and nutrition sensitive social protection (NSSP). JHU and IFPRI team members were also involved in the Improving Coverage Measurement and Improve projects, which focused on validation of health and nutrition intervention coverage indicators collected through household surveys.
Based on our cumulative experience, we are proposing a systematic approach to the development and validation of intervention coverage indicators that is rigorous, cost-effective and timely. It is a three-step process that includes landscaping, formative research, and quantitative validation. We focus on household survey indicators but expect that the core approach is also relevant to design and validation of administrative data tools.
Step 1: Landscaping to engage data users and identify relevant knowledge
This step identifies the intended uses of the indicator, potential indicator specification issues, and relevant research and gaps. Consistent with the nutrition data value chain, indicator prioritization should be rooted in the needs of data users. Engaging with data users is necessary to understand how the proposed indicator might be used and what information it needs to capture. For example, when developing coverage indicators for micronutrient supplementation in pregnancy, do stakeholders need qualitative or quantitative measures of adherence (e.g. regularly took pills vs. at least 180 pills), over what period (e.g. last month vs. entire pregnancy), and for which supplements (e.g. any iron containing supplement vs. specific to IFA and MMS)? This user engagement may be accompanied by a desk review of published and unpublished literature to learn from relevant measurement work.
Step 2: Formative research to design question and visual aids
Formative research that engages the respondent population is essential to guide question formulation. The aim is to identify intervention design features and terminology that are widely recognized and understood by the respondents to include in questions and visual aids. The first stage of formative research may include field visits to observe implementation and interviews with the respondent population and frontline workers. This stage seeks to identify how the intervention is delivered and how the target population/respondent experiences the intervention. Continuing with the micronutrient example, this would include what micronutrient supplements and medications are available to pregnant women, and whether and how women distinguish IFA from MMS tablets.
The second stage of formative research is cognitive interviewing, a qualitative approach that explores how respondents interpret and respond to survey questions. Draft questions should undergo iterative cognitive interviewing to refine wording and identify the most promising questions for quantitative validation.
Step 3: Validation of proposed questions
As a final step, a criterion validation study compares a respondent’s answers to a “gold standard”, an objective measure of whether the person received the intervention. These studies assess whether the proposed indicator and questions provide accurate measures of intervention coverage. You can learn more about criterion validation methods here. A gold standard measure can be complex and expensive to obtain, however. As a result, many coverage indicators have limited or no validation data.
How can this be done in a cost-effective and timely way?
Nesting indicator development and validation activities within intervention trials and implementation research (IR) studies will save money and time compared to independent measurement research. These research activities often include formative or pilot phases which can accommodate key informant interviews and cognitive testing. Intervention trials, and some IR studies, also collect gold standard data on whether an intervention was received by a study participant. It is relatively easy and inexpensive to add a criterion validation study by posing recall questions to study participants during follow-up visits and comparing their responses to the trial’s existing gold standard data.
DataDENT is applying elements of this approach to the development of new indicators and household survey questions for coverage of micronutrient supplements during pregnancy. We are trying to address some of the validity issues with the iron supplementation questions currently used by Demographic and Health Survey (DHS) and other large-scale surveys. We have leveraged available resources to carry out different steps of the process in different contexts. Ideally, we would implement all three stages in the same context.
To date, DataDENT has talked with data users in the Healthy Mothers Health Babies Consortium about whether it is important to have separate estimates for IFA and MMS coverage. We are conducting formative research in Ethiopia and India, to identify which supplements are available to pregnant women at health centers and retail locations, and what terms and visual cues women use to distinguish between different supplement types. Questions and visual aids are then drafted and refined through iterative cognitive testing.
We are actively seeking groups carrying out MMS IR who could support further cognitive testing and nested validation studies. One promising opportunity is an IR trial being conducted by Hellen Keller International-Nepal that is recording the number of MMS distributed to study participants and conducting pill counts and interviews at different points during pregnancy and after birth. It is possible to use these existing data to compare women’s responses to questions about MMS received and consumed with gold standard study records and pill counts.
Call to action
Nesting rigorous coverage indicator development and validation within trials, evaluations, and IR should be more intentional and common. Now is the time to start developing coverage indicators and data collection tools for calcium and balanced energy and protein (BEP) supplementation during pregnancy and small-quantity lipid-based nutrient supplements (SQ-LNS) for children. Researchers, donors and governments can work together to prioritize development of coverage monitoring systems by leveraging these opportunities to design and validate coverage indicators in a timely, cost-effective way.