Towards a culture of evaluation and evidence-based decision making in global health
Reflections on THET Conference 2016: Evidence, Effectiveness & Impact
Last week we attended THET’s annual conference in London. As researchers specializing in qualitative and quantitative methodologies, the theme of evidence, effectiveness and impact was a particularly relevant one for us. Monitoring and evaluation are increasingly being recognized as critical components in establishing the impact of any global health intervention. It was interesting to gain an understanding of the latest thinking in this area from a health partnership perspective. Health partnerships are collaborative relationships between organizations and institutions working to improve the health of people living in developing countries. Though partnerships take various forms, they involve working together to achieve common goals and meet specific needs. Involvement of in-country organizations ensures the approach is ‘country led’, while involvement of other countries enables skills transfer and knowledge exchange.
"Generating robust evidence about health partnerships in low and middle income countries is fundamental to informing future decisions."Generating robust evidence about health partnerships in low and middle income countries is fundamental to informing future decisions. Donors and policy-makers are increasingly demanding this evidence. As such there is a growing need to demonstrate the impact of interventions for the dual purposes of monitoring and motivating. However health partnerships are complex, interacting, evolving, context-dependent phenomena, as the morning panel discussion acknowledged. This makes them challenging to evaluate.
In the breakout session I attended, I was interested to hear about some of the partnership contributions to reproductive, maternal and child health in Uganda, a country where, despite improvements, infant mortality remains alarmingly high. Much of this mortality is in neonates, and we heard from partnerships involved in implementing the WHO ETAT training program to establish best practices in hospital care. In addition we heard about uSHAPE, a Sexual Health and Pastoral Education program. Common challenges for these programs include cascading the education to the community level via ‘train the trainer’ outreach in a sustainable way, and overcoming cultural resistance to providing this education, from both a gender role and a religious perspective.
The Hon. Joyce Moriku Kaducu, Uganda’s State Minister for Health, welcomed the numerous partnerships underway in her country, both on-going and planned. However she flagged the challenge of limited manpower and stated the desire for programs to focus on being able to make a difference within the current infrastructure. The desire is for interventions to be designed to work within the current framework and be able to demonstrate their positive impact. Sustainability is key here: ensuring the country is able to maintain progress once grant funding has ended or after the partner’s involvement in the program has concluded.
Ugandan State Minister for Health Dr Joyce Kaducu with delegates
One of the challenges raised in several sessions was the difficulty of attracting funding for evaluation – as it is the least ‘sexy’ part of the intervention from a donor’s perspective. However, as Dr. Paula Barasiter of Kings College London argued in her presentation, there is a need to build this culture and invest in evaluation to prevent the ‘castles’ of health partnerships from falling down when they are scaled up.
It was encouraging, though, to see in spite of funding challenges that research, both qualitative and quantitative, is already being used to evaluate many of the partnership interventions discussed during the conference. Typically this research took the form of small scale qualitative in-depth interviews and/or more structured self-completion interviews to understand and measure behavior. A case study of this was the Ghana Stroke Care study, which included 6 semi-structured in-depth interviews with follow-up planned for late 2016. However monitoring the impact of any of these partnership interventions in a robust way is fraught with difficulty. Observer bias effect (the risk that people observing the implementation increase the likelihood of changes being implemented during the observation period but not outside of it), small sample sizes and competing agendas of different stakeholders are just some of the challenges.
Some of the key academic principles that were discussed as ways to help overcome these challenges form part of the typical project process used by the Research Partnership when we conduct research. From our expertise through experience, we recognize it is essential to achieve buy-in from all stakeholders at the initial definition phase, carried out at the very start of the process, not just when presenting the results of the evaluation. "...we recognize it is essential to achieve buy-in from all stakeholders at the initial definition phase, carried out at the very start of the process, not just when presenting the results of the evaluation."Use of behavioral science theory, alongside measurable outcomes, can provide a scientific basis on which to inform future endeavors. As an independent research organization not involved in implementing the programs we monitor and evaluate, we are less impacted by observer bias.
Another recurring theme that I encountered throughout the day was discussion about WEIRD (Western, Educated, Industrialized, Rich, and Democratic) countries, and the point that data/thinking/attitudes from WEIRD countries did not necessarily apply to developing countries. Moreover, even within a region, there was a recognition that it’s important to remember not all developing countries are the same; we should not automatically assume that what has successfully been implemented in Country X will work in Country Y. The implication being that program designers must immerse themselves in how things are done and what can realistically be achieved within this framework. This underlines the importance of primary research, not just to evaluate interventions once they have been implemented, but pre-intervention to ensure they are grounded in the local context.
A case study in point is the issue of ‘graveyards’ of well-intentioned interventions that are poorly executed. An example cited was that of expensive medical equipment delivered to low income countries by donors who, though clearly well intentioned, fail to recognize that without adequate training, the recipients will not be able to use them and they will be left to gather dust, rather than making the difference the donors intended. In contrast, some of the companies that exhibited at the THET conference specialize in medical technology for limited resource environments – designed for simplicity (limited training needed, easily usable and sustainable) and reliability (e.g. usable in the case of power outages, which unfortunately remain all too common in LMIC field hospitals and disaster situations). One example on show was Diamedica’s oxygen concentrator driven CPAP device.
Are there better ways for those behind global health interventions to analyse the impact of what they do? As a third party independent research company, we feel we are well placed to offer support in evaluation because this is something we are already doing for healthcare companies’ research needs. We offer impartiality, sophisticated methodological approaches, expertise in question design and the ability to liaise with all stakeholders without any agenda beyond providing objective, robust and rigorous evaluation. In addition we can provide unbiased recommendations for optimization and improvement. Global health organizations don’t always have the in-house capacity and capability they need to monitor and evaluate the interventions they are involved in, so bringing in private sector approaches such as ours can be an effective solution.
Sign up to receive Rapport
Rapport is our e-newsletter and online resource for sharing our expertise and experience in global healthcare market research. Sign up here »