The practice of evaluation as an evaluation of practices

This article describes the theoretical basis and characteristics of a process-based, negotiated and generative approach to the evaluation of organizations. Illustrated by a case study, it highlights the inappropriateness of standardized instruments in the face of complexity and uncertainty, suggesting ways to improve the practice of evaluating dynamic complex systems. It underlines how conducti…

Taking stock of complexity in evaluation: A discussion of three recent public…

Edition

Volume 21, Number 4 October 2015, pages 481-491

ISBN/ISSN

1356-3890

Collation

-

Series Title

Evaluation

Call Number

-

Theory in evaluation: Reducing confusion and encouraging debate

One of the goals of the journal Evaluation was to advance theoretical and methodological understandings of evaluation. Twenty years later theory as understood in the field of evaluation is fragmented, a problem widely recognized in the evaluation community. The goal of this article is to make a modest contribution to clarifying this situation. We first present two typologies of evaluation theor…

Future-oriented impact assessment: Supporting strategic decision-making in co…

The article proposes a systemic and future-oriented evaluation approach designed to support decision-making in complex socio-technical environments. The approach integrates established methods of evaluation, foresight, impact assessment, system dynamic modelling and societal embedding within a single framework to provide versatile information to increase strategic intelligence in decision-makin…

The contribution of process tracing to theory-based evaluations of complex ai…

This article focuses on methodological challenges in evaluating complex program aid interventions like budget support. We show that recent innovations in process-tracing methodology can help solve the identified challenges and increase the strength of causal inference made when using case studies in demanding settings. For the specific task of evaluating the governance effectiveness of budget s…

Using theories of change in the CGIAR Research Program on Agriculture for Nut…

Theories of change are increasingly being discussed and referenced in development evaluation even while the elements of what a theory of change consist of differ widely among applications. Equally, examples of actual use of theories of change other than as overview illustrations of interventions are rare. In this article, we present generic models of theories of change for both straightforward …

Integrating quasi-experimental and inductive designs in evaluation: A case st…

Evaluations of ‘natural experiments’ in public policy are typically considered ‘weak’ evidence. Challenges include: making credible claims for causal inference (internal validity); generalizing beyond the case (external validity); and providing useful evidence for decision makers. In public health, where experimental evidence is encouraged by funders and enjoys a degree of rhetorical favour, in…

Towards a comprehensive framework for the evaluation of small and medium ente…

This research demonstrates the relevance of the evaluative cycle and its diverse methodological designs in small and medium enterprise (SME) policy. We structure our arguments based on the most common phases of the cycle, namely policy justification, needs, policy theory, implementation, impact and efficiency assessments. We use an in-depth case study of public assistance to an SME to illustrat…

Developing expertise in managing dialogue in the ‘third space’: Lessons f…

Where two or more persons from different professions meet to discuss a shared project, a dialogic ‘third space’ is opened up. Such dialogues (or multilogues) can be problematic for reasons which are well attested to in the literature. Third spaces can be sites of hostility and defensiveness, or of creativity and learning. This article explores the nature and management of the third space, notin…

A holistic, learning-centred approach to building evaluation capacity in deve…

There is growing pressure on development organizations to improve their evaluation systems and capacities. This presents considerable challenges for time- and resource-poor organizations in developing countries. Evaluation capacity development (ECD) approaches are needed that are appropriate and effective for such organizations. We argue that this requires a long-term, holistic, participatory, …

Aspects of the institutionalization of evaluation in Finland: Basic, agency, …

This article draws on neo-institutional theoretical ideas to empirically examine the institutionalization of evaluation in the national government of Finland. The results indicate ambiguity in the basic institutionalization of Finnish evaluation, and imprecision in the agency of the actors that carry out or commission evaluations or utilize the evaluation results. Some Finnish institutional pra…

Policy experiments: Investigating effectiveness or confirming direction?

In England, ‘policy experiments’ are largely synonymous with the use of randomized controlled trials (RCTs) to test whether one policy ‘works’ better than another. While advocacy of the use of RCTs in public policy presents this as relatively straightforward, even common sense, the reality is different, as shown through analysis of three high profile policy pilots and their evaluations undertak…

From mechanism to virtue: Evaluating Nudge theory

Ever since Thaler and Sunstein published their influential book Nudge, the book and the theory it presents have received great praise and opposition. Nudge theory, and more particularly, nudging may be considered an additional strategy providing some novel instruments to the already rich governance toolbox. But what is its value? The current debates on Nudge theory are often highly normative or…

The strategic ritual of irony: post-Tiananmen China as seen through the ‘Pe…

Inspired by Tuchman’s concept of the ‘strategic ritual of objectivity’, we argue that journalists employ what can be called the ‘strategic ritual of irony’ in their accounts to convey moral stance toward morally ‘tainted’ stories, often under the façade of objectivity. Systematic reading of American journalists’ memoirs and writings reveals that their portrayals of post-1989 China, against the …

Stop griping, start complaining: how public discontent can trigger a change i…

This study examines the conditions under which public manifestation of discontent can lead to changes in the broadcasting of offensive media content. Our analysis is based on ethnographic research within the ombudsman at a regulatory institution in Israel, on content analysis of 2142 complaints regarding perceived inappropriate broadcasted content between the years 2005 and 2010, and on an exam…

Broadcasting to reflect ‘life and culture as we know it’: media policy, d…

The possibility that broadcasting powers could be devolved from the UK government to the Northern Ireland (NI) Assembly has been a matter for political discussion in NI. During a 2013 debate in the Assembly on the matter, the dominant Irish nationalist political party called for such devolution to increase the levels of programming that reflect ‘life and culture as we know it’. The subject is d…

Pro-anorexia/bulimia censorship and public service announcements: the price o…

Individuals, particularly women, are fixated on weight loss, driven by the goal of achieving a ‘skinny’ female physique that is desirable in western/ized cultures. There are online forums where individuals refer to themselves or their eating disorders as ‘pro-ana’ and ‘pro-mia’; their posts on these sites both align with, and challenge, what medical and mental health professionals define as ser…

The exercise of symbolic power by women’s magazines from the 1960s to the p…

This study analyses alcohol-related articles appearing in Finnish women’s magazines from the 1960s to the present day. Women’s magazines are approached as institutions constituting feminine publicities that address issues of interest as well as problems and contradictions in women’s everyday life. Influenced by Pierre Bourdieu’s concepts of field, habitus and capital, data analysis focuses arou…

Losing focus: goal displacement at an alternative newspaper in El Salvador

This study examines whether a Salvadoran alternative newspaper maintained its critical, independent, and alternative position after the country’s first leftist president was elected and the newspaper no longer was in opposition to the government. Via a content analysis and in-depth interviews that drove the content analysis, this study improves our understanding of ‘alternativeness’ in a non-US…

Media encoding in science fiction television: Battlestar Galactica as a site …

We examine the television show Battlestar Galactica (BSG) through interviews with creative people working on the show to illustrate the production context of the show and the science fiction (sf) genre. Media scholars suggest sf stories are critical stories about our political systems and our anxieties about new technologies, social change, race, gender, class, and religious conflicts. We inves…