INTERVIEW WITH RICK DAVIES

Rick Davies has acquired an international following through his website “Monitoring and Evaluation News” (http://www.mande.co.uk). The site and its associated email list connect evaluators in far-flung countries and offer links to a wealth of evaluation resources. An independent consultant based in the United Kingdom (UK), Rick is also an innovator, having developed the now widely-used tool known as "Most Significant Change" (MSC), as well as a simple tool for measuring poverty. Below is the full transcript of an AEA interview with Rick that took place in May 2008:

1. By way of introduction, please briefly describe your current position, as well as your responsibilities. When and how did you get involved in the field of evaluation?

I am an independent M&E (monitoring and evaluation) consultant, based in Cambridge, UK. I work with NGOs (non-governmental organizations), bilateral and multilateral aid organizations, in Africa and Asia (most recently Ghana, Uganda, Indonesia, Vietnam). I began doing this kind of work in 1990, after 10 years of working for aid organizations in Africa and Australia. I also started my PhD thesis on organizational learning in NGOs at the same time. One advantage of doing the PhD part time, while continuing consultancy work, was that it enabled both activities to learn from each other.

2. How would you describe your role as an evaluator? How would you describe the role of M&E vis-à-vis evaluation?

I try to find contracts that allow me to be engaged with projects on a visiting basis over a period of years, rather than seeking one-off contracts for stand-alone evaluations. In Ghana I was helping build the M&E capacity of an NGO funding mechanisms that was supported by multiple donors[1]. In Uganda my role is different; my six monthly visits are a form of meta-monitoring of a rural development project which has its own M&E systems[2]. In Uganda I am contracted by the donor to the project. In Ghana I was contracted by the project. In Indonesia I am involved in a yearly review of a small number of maternal health projects, leading a small team which includes maternal health specialists. This is more like an external evaluation process than meta-monitoring and involves a more complex set of stakeholders, who are trying to coordinate their work along the lines proposed by the Paris Declaration. In the UK I have a different role again, providing technical advice to two M&E specialists within a large grant making NGO, so here I am effectively being a consultant to consultants. They are trying to improve the way their organization learns from its grant making experiences.

Overall, my roles are diverse, but with a preferred emphasis on meta-monitoring and associated capacity building rather than evaluation. However I do feel I need to refresh my experience of external evaluations. On the other hand, I am also wary, knowing how frustrating these events can be because the serious time constraints that are usually imposed.

3. What things have happened in the international evaluation scene over the last few years that you have found to be particularly interesting and exciting? In what ways have you been involved in them?

I am not sure I have the overview that is needed to answer this question.  But there are two widely discussed developments that are of interest to me. One is the push from some quarters for more rigorous approaches to evaluation, including the use of experimental designs. My first degree, in experimental psychology, left me with a continuing interest in research methodology. Combined with my experience of many evaluations of development aid being conducted under less than ideal circumstances I am very sympathetic to the demand for more rigor. On the other hand, the larger scale and institutional complexity of many development aid programs makes any notion of planning a with-without / before-after comparison (even without randomized treatments) seem like a fantasy. In practice, over the last twenty years I have only been able to do one with-without/before-after comparison using quantitative data, and that was almost by accident[3]. What I do try to promote is more use of internal comparisons within programs, because often there can be significant variations in the nature of interventions across locations (e.g. districts) and groups. There are echoes here of Pawson and Tilley’s Realistic Evaluation – even in a project that seems to have “failed” overall there can be important localized successes that are worth finding and learning about.

The other development of interest to me is the attention being paid by international aid donors to harmonization of their aid efforts, and alignment with host government policy, as expressed in the Paris Declaration[4]. This is increasing the interest in jointly conducted evaluations involving multiple donors and other stakeholders, including the kind I am involved with in Indonesia. For me, the experience has been like attending a Summer School in Complexity Theory. Even at the humble level of identifying an appropriate time for an evaluation, the interaction of all the constraints on all the actors’ available time quickly closes down many options. It is simply not a scalable response to the need for more harmonized evaluation work amongst multiple aid agencies. The experience implies the need for more strategic (i.e. selective) thinking about who should be involved, and other quite different ways of working in large complex systems with many autonomous actors. More on this below.

4. From an international perspective, what particular challenges do you see being faced in the way in which the practice of evaluation is evolving? What is being done to address them?

I think there are two developments that concern me. One is the very substantial expansion in international development aid budgets that is not being accompanied by a similar expansion in resources devoted to their management, including evaluation. DFID (the UK's Department for Independent Development) is a classic example. The members of DFID’s recently formed Independent Advisory Committee on Development Impact have expressed the same concern[5]. Associated with this development has been the increasing proportion of bilateral aid that is being spent via Direct Budget Support (DBS) to governments that are believed to have some minimal level of competence. In these circumstances donors must, in the first instance, rely on the recipient governments own M&E capacity to know how government budgets are being used. They also need to help build those governments’ M&E capacities, and to meta-monitor that capacity. My experience with the Ghana Poverty Reduction Strategy has highlighted how limited government capacities can be, how hard it can be to develop that capacity, and how limited donors’ interests are in a critical review of that capacity. For many international donors, the use of DBS is administratively convenient and ideologically comfortable. This limits the type of evaluation findings that are likely to be listened to.

5. Have you been involved in creating methods that you think could be more broadly used in other areas of practice or other parts of the world?

My PhD thesis[6] was unusual in that it led to the development of a method of “monitoring-without-indicators” that is now widely used in many countries around the world, both rich and poor. That is the “Most Significant Change” technique[7]. My main problem is not making sure more people use the method, but working out how I (and others) can keep up to date with, and learn from, the many different ways this method has been used. The method (and the thesis) was based on an evolutionary epistemology, heavily influenced by Donald Campbell and Gregory Bateson. Speaking within that framework, it is the many emerging variants of MSC that I now need to subject to some of my own selection!

Also arising from my thesis research are two other qualitative methods, with the same epistemological roots, that I occasionally refer to as the poor neglected cousins of MSC. One is Hierarchical Card Sorting[8], which is current being used by a British NGO to clarify the implicit strategy within each of their various portfolios of projects. The other is Evolving Storylines[9]. This is a participatory process for evolving multiple competing storylines about the future, or the past. It has only been used once, to my knowledge, with school kids in Wales. I think it could be very helpful for getting organisations to think more creatively about their future strategy, at the corporate or program level.  

In more recent years I have been exploring and promoting the use of network models. This is not a particular evaluation method, but rather an alternative way of describing intended and actual developments. My assumption here is that if we have better ways of describing the world then we may be in a better position to express and test our theories about the world. My second assumption is that many evaluation settings are best seen in terms of networks of many actors with some degree of autonomy and no single centrally commanding authority governing them (i.e. not as a hierarchy). The descriptive tools (network diagrams, network matrices and measures, and supporting software) used for constructing network models all come from the field of Social Network Analysis.

There are however three challenges in promoting the use of network models. One is to get people thinking in terms of networks as a kind of a base metaphor, in the same way that in the past people may have seen clockworks as a base metaphor for how the world works. The second is to sift through what often appears to be a surfeit of technical capacity to analyze networks, in order to focus in on the simplest and often most useful basics. The third challenge is to develop more participative and interpretative approaches to the description and analysis of networks, in contrast to the number crunching nature of much social network analysis in the academic world. This is all very much a “work in progress”.

There is another area where I have not been creating methods, but where I have been helping others talk about methods. That is the MandE NEWS website[10], which I originally set up with the help of Oxfam and other NGOs in 1997, and the associated MandE NEWS email list[11], which was set up some years later. The email list now has more than 1700 members worldwide, which makes it one of the largest in the field. It also has a noticeably high proportion of members from Africa and Asia, which is great to see. The email list generates quite a lot of content for the website and the website (with Google’s help) leads a lot of people to the email list. The immediate challenge with the email list is “patrolling the borders”, stopping off-topic postings and making sure the conversations across multiple cultures remain friendly and constructive. Overall I continue to be surprised how helpful people on the email list can be to others, especially when some don’t seem to have done much prior “homework!” The more interesting and longer term challenge is to identify where the email list and website best fit within an increasingly large and complex ecology of other websites and email lists dealing with monitoring and evaluation issues.

6. What should be the international field's top priorities with regard to evaluation policies or strategies over the next few years?

Promoting transparency: transparency of aid processes, of evaluation processes and evaluation results. Lack of transparency can be an indicator of ineffectiveness, in that those responsible for corrupt and / or ineffective practices are not usually keen for these practices to be made publicly visible. On the other hand, the presence of transparency can be an instrument for increasing effectiveness, because as they say in the world of software design, “with enough eyeballs, all bugs [problems] are shallow.” Overall, the aid process needs to be much more public. One positive example is the Guardian newspaper’s website and blog on AMREF’s (African Medical & Research Foundation's) Katine project in Uganda[12]. This is the most publicly exposed project I have ever been associated with, and one which I hope we will learn many lessons from. It won’t be an easy ride. It will be much more complex than an open source software because the objects of concern are peoples lives, and the politics that govern them, not inert lines of computer code.

Increasing expectations: My other proposed priority, which I would like to see widely accepted, is that aid projects must not only have an impact and be able to identify that impact. They must also be able to communicate to others how to replicate that impact. They must produce a usable “public good” (in the form of useful how-to-do-it information, not anodyne “lessons learned”), in addition to having a significant local impact. Only this way will aid investments ever hope to have an impact that corresponds to the scale of the problems being addressed. That knowledge production process will be aided if there is increased transparency about aid processes: there will be more eyeballs looking at the problems and the proposed solutions, as the work progresses.

Looking in to the future I think there needs to be a sea change in attitudes change about transparency. Aid programs are funded by public monies, either from taxes or donations. Therefore there is a prima facie argument that by default most information about the use of those aid monies should be available in the public domain, and that is the exceptions that need justification. Instead we have the opposite situation whereby most information is not in the public domain, that requests for public availability often need to be argued for, and that much information that is publicly available is there as the result of decisions made by people in charge of public image management within their organization. If “the eyes are the window of the soul” of individuals then perhaps websites are the windows of the soul (or lack thereof) of organizations. This is where I would like to see more monitoring and evaluation effort by independent parties. It should not be too difficult.

7. Thank you for your responses. Is there anything else in particular that you would want to share with AEA members?

My “Rick on the Road” blog[13], where many of these ideas have been aired, and where there will be more to come.

And some reading that may be of interest:

Benkler, Y. (2006)  The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press.

Bruns, A. (2008) Blogs, Wikipedia, Second Life and Beyond: From Production to Produsage.  Peter Lang.

American Evaluation Association│16 Sconticut Neck Rd #290 │Fairhaven MA 02719
1-888-232-2275 │1-508-748-3326 │info@eval.org